I’m beginning to doubt that my current approach of transporting data from a source to a sink by using threads for each source will ever work. It appeared to be way too simple, too!
First I thought that processing data and metadata (the format of the data, among others) asynchronously should be fine; however, it needs to be possible to process different formats by the same pipeline: you do want to be able to play an MP3 file followed by an Ogg Vorbis file, don’t you? Yes.
So I rewrote the transport code to transport data and metadata in the same stream, using packets that always contain data and sometimes contain (maybe new) metadata. But I think I still have the problem that some thread down the pipeline has not yet finished getting all data from its source when the metadata changes are propagated.
The next think I’m going to try is to cut down the number of threads I’m using. It should be possible to use a single thread for a pipeline (under the assumption that the pipeline doesn’t split — which it will have to, later on). This probably needs some rewrite of how sources expose their data because I need to be able to perform non-blocking reads which Java’s InputStreams usually don’t do.