VideoTools

VideoTools is an amateurish experimental collection of Swift classes for compressing, and decompressing H264 video.

Decompressor

Decompressor decompresses (duh) an Annex B H264 byte stream, if you don't know what this is, you are probably in the wrong place. I know I was! Never fear! This is a common H264 network stream format and it more or less looks like this :

[ start_code (3...4 bytes), NALU header (1 byte), NALU stuff (n bytes) ]

Usually (lol) you will get packets wrapped in some kind of header that indicates length, etc. You will need to process this and extract the packet before feeding it to the decompressor. Additionally, if you have multiple packets, you will need to extract them separately, the decompressor expects single NALU packets. You should be able to use the start code (either 0x000001 or 0x00000001) to split them up

Anyhoo ...

import VideoTools

let decompressor = Decompressor()


// set up the handler

decompressor.handler = { [self] result in
  
  switch result {
    
    case .error  (let error)  : print (error)
    case .format (let format) : print (format)
    case .frame  (let buffer) : // do something with decompressed frame
  }
}

// ...
// ...
// get your compressed data from somewhere
// ...
// ...

decompressor.process ( data )

As you can see the handler will get one of three (ish) things, a CVPixelBuffer of decompressed video, a CMFormatDescription when the format is first created or when it changes (e.g. the resolution, frame rate or orientation, etc change).

Or an error. There are lots of these.

SampleBufferConverter

Now that we have our have our decompressed video frames in CVPixelBuffers, we would, probably, like to display them somewhere. It is possible that whatever we want to display them with (e.g. a AVSampleBufferDisplayLayer) will want CMSampleBuffers. Awks.

import VideoTools


let buffer = // ... get CVPixelBuffer from somewhere

if let samples = convertor.make (from: buffer, pts: CMTimeMake(value: frame, timescale: 30)) {
  // do thing with samples
}

NB that you should definitely calculate a proper CMTime for your samples, this is left as an exercise for the reader because I have grown to loathe CMTime

If you are using this library and it is not working, like all you see is blank output, I guarantee you your problem is not having correct CMTime

CMTime - Briefly

CMTime docs are here and I'm not going to do a big write up because I'm not at all certain that I'm getting this right but essentially if you do CMTimeMake ( value: frame, timescale: FPS ) where frame is a monotonically increasing count and FPS is, well, your FPS it will probably work. There's a lot of fannying about with multiples in the wild, so go read up on it.

Compressor

Doesn't quite do the opposite. Compressor takes a stream of CVPixelBuffer and compresses them, but by default, this will not be an Annex B stream but an AVCC buffer. We'll get to that in a moment.

import VideoTools

let compressor = Compressor ( width: 1280, height: 720, realtime: false, seshprops: [:] )

compressor.handler = { result in
  switch result {
    case .failure (let fail)   : print (fail )
    case .success (let buffer) : // do something with your compressed CMSampleBuffer
  }
}

// ...
// ...
// get your CVPixelBuffers from somewhere ...
// ...

compressor.compress ( buffer, pts: .invalid )  // don't do this, provide a CMTime 

As you can see me must provide the compressor with a resolution, so if your input format changes, you will need to kill the compression and build a new one.

The realtime parameter determines whether the compressor is required to operate in real time. Highly recommend that if you are writing to a file that you set this false as otherwise the compressor will rush and do things like drop frames if it can't encode them at the FPS.

The seshprops param takes a dict of type [CFString : CFTypeRef] containing kVTCompressionPropertyKey... keys and values for advanced tweakage, these are applied after the defaults. You will need to go and look these up in the Compression Properties documentation.

AnnexeBPacketizer

So now we have compressed some video there are a few things we might want to do with it. The first of these is to send Annexe B packets over a network to do this we use the Packetizer.

The packetizer is very simple.

import VideoTools

let compressor = Compressor ( width: 1280, height: 720, realtime: true, seshprops: [:] )
let packetizer = AnnexeBPacketizer()


// ... insert you buffers to the compressor ...

compressor.handler = { result in
  switch result {
    
    case .failure(let fail)   : print(fail)
    
    case .success(let buffer) : 
        
        let packets = packetizer.packetize ( buffer )
        // ...
        // do something with your packets 
        // ...
  }
}

What you do with those packets is of course up to you, you will likely need to wrap them in some kind of header frame with (e.g.) length and other deets. How you do that is up to you though.

OutputCompressor

Not really recommended but if you just need to quickly write to a file and you know that your resolution, FGPS, etc will not change, then OutputCompressor will wrap up a Compressor and a AVAssetWriter and get you going.

I highly reccomend that you write a custom output chain, but this will get you going.

import VideoTools

let capture    = // ... some way of getting CVPixelBuffer
let output     = OutputCompressor ( width: 1280, height: 720, fps: 30 )

try output.createFile ( url: URL ( fileURLWithPath:"/YOUR/PATH/test.mov" ), overwrite: true )
try output.start()

capture.handler = { result in
  switch result {
    // ...
    case .success (let buffer): if output.isReady { try! output.enqueue ( buffer ) }
  }
}

// ... some time later ...

output.finish {
  print("all done")
}

Note that the output compressor will throw things at you. Note also that there is a second function public func enqueue ( _ pixbuff: CVPixelBuffer, pts: CMTime ) throws that you should use if you know what your CMTime should look like.