Skip to content

Development

Jump to bottom Edit New page
sofian edited this page Jan 9, 2013 · 16 revisions

Basics

I would like to make the list of the basic gears and features that are missing in Drone (and to classify them). I will do this by browsing through the:

  • our old wiki
  • PtolemyII
  • Isadora
  • Quartz composer
  • Puredata

Gears

OSC:

  • "OscExtract" (or a better name) Takes an osc message as an input and outputs the path and arguments as a list

Frei0r:

  • Make GearFrei0r work again

Math:

  • Values
    • Generators
      • Random (uniform, normal, poisson, bernouilli, white noise, pink noise, etc) (cf. PtolemyII has a bunch of Random gears)
    • Unary operators
      • stateless
      • -x
      • trigonometric (sin, cos, tan, etc...)
      • exp, pow, log, sqrt, sign, etc.
      • time-based (moving):
      • moving max / min
      • moving avg (exponential)
      • moving diff (difference between current input and last)
      • accumulator (computes the sum / avg / stddev of received inputs, with a reset input to reset it)
    • Binary operators
            • /
      • max / min / clamp
    • Expression
  • Logic
    • Conditions: == != < <= > >=
    • Operators: ! && || XOR
  • List
    • Unary operators
      • max / min / avg / stddev
    • Binary operators
            • /
      • max / min / clamp
      • dotproduct
      • sum
    • Expression
  • Control
    • For
    • Gate
    • Select (or switch)

Visualisation:

  • Oscilloscope: converts signal into video
  • FFT display: converts signal into spectrum bands / possibly two gears: fft -> array2histogram ( -> videooutput)

Time:

  • Clock: current time / program time / metagear time
  • Sleep: Interrupts stream for a given time upon activation

Features

  • Dynamic plug in / plug out: In some cases you want to have a property that influences the number and types of plugs.
    • This is the case of (for example) the list pack/unpack gears.
    • There are issues though: for example what happens if a property removes a plug that was plugged into? (it could just remove the link, at least as a first step)
  • (later) Real-time audio processing (with associated gears for sound generation and playback)
  • (later) More tunable mechanism for schedulers (each gear could have a scheduler, with video/audio by default but the possibility to configure and add other schedulers easily)

Bugfixes

  • The mechanism for sleeping() plugs is not 100% working
    • First of all there's a bug: sleeping(false) is not propagated instantaneuously from runVideo() (if you set a plug to sleeping(false) in runVideo whereas it was sleeping(true) before, the following gear won't execute until at the next loop)

Types

  • StructType (together with associated gears)
  • (later) FunctionType (lambda-calculus)

Cross-platform video

Ideally we want to find a tool that does everything on every platform. Practically, we would be happy with multiple tools, as long as they are cross-platform (eg. one tool for video reading, another for capture).

Library Description Video cap. Video source
Myron/WebCamXTra That's the one used in Processing for video capture Yes No
Portvideo (latest version is here Cross-platform video capture (Linux, Mac, Windows). That's what Myron/Webcamxtra is actually using. Yes No
OpenCV HighGui Well-known cross-platform computer vision software (see reference with example here Yes ???
OpenML Cross-platform media library Yes ???
ReacTIvision Software for augmented reality applications (we could strip the code) Yes ???
GStreamer Cross-platform library for media No sure (With v4l under Linux, not sure about other platforms) Yes
QtMultimedia (en français) and Phonon (see also Phonon main page) Cross-platform media library in Qt (which we are already using). Phonon is the high-level module for QtMultimedia. However it might look too good to be true. No sure. There is this "Phonon capture API" that looks interesting but it's from 2010 and seems to work just on Linux. Yes

Theory / references

Perhaps understanding better the theory of dataflow programming would allow us to make a better software.

Gear signatures

A Gear can have many different signatures. A signature is simply a set of plug-ins and plug-outs.

Example 1 : A gear that defines an input plug that could be one of two types (video or texture).

Gear_MyGear::Gear_MyGear(Schema *schema, std::string uniqueName) : 
  Gear(schema, "MyGear", uniqueName),
  _droneQGLWidget(NULL),
  _window(NULL)
{
  getDefaultPlugSet()->addPlug(_VIDEO_IN = new PlugIn<VideoRGBAType>(this, "IN", true)); // equivalent to addPlug(_VIDEO_IN = new PlugIn<VideoRGBAType>(this, "IN", true));

  addPlugSet(new PlugSet(this, "texture"))->addPlug(_TEXTURE_IN = new PlugIn<TextureType>(this, "IN", true));
}

...
void Gear_MyGear::runVideo()
{
  if (getCurrentPlugSet()->name() == "texture")
  {
    doSomethingWithTexture(_TEXTURE_IN->type());
  }
  else // default
  {
    doSomethingWithVideo(_VIDEO_IN->type());
  }
}

Example 2 : A gear with two signatures, both of which share a common ValueType.

Gear_MyGear::Gear_MyGear(Schema *schema, std::string uniqueName) : 
  Gear(schema, "MyGear", uniqueName),
  _droneQGLWidget(NULL),
  _window(NULL)
{
  _SCALE_IN = new PlugIn<ValueType>(this, "Scale", true);
  getDefaultPlugSet()->addPlug(_VIDEO_IN = new PlugIn<VideoRGBAType>(this, "IN", true));
  getDefaultPlugSet()->addPlug(_SCALE_IN);

  addPlugSet(new PlugSet(this, "texture"));
  getPlugSet("texture")->addPlug(_TEXTURE_IN = new PlugIn<TextureType>(this, "IN", true));
  getPlugSet("texture")->addPlug(_SCALE_IN);
}

...
void Gear_MyGear::runVideo()
{
  if (getCurrentPlugSet()->name() == "texture")
  {
    doSomethingWithTexture(_TEXTURE_IN->type(), _SCALE_IN->type()->value());
  }
  else // default
  {
    doSomethingWithVideo(_VIDEO_IN->type(), _SCALE_IN->type()->value());
  }
}

Technical notes

Add a custom footer
Add a custom sidebar

Clone this wiki locally