Reintegrate projectM Visualizer

[Gustavo L Conte]

I just noticed now that the presets that seem not to work with these new conditions, actually fade to black only when selected in the interface, but when they run as a playlist, they DO WORK AS EXPECTED. I've just selected a few presets that work and doesnt work, and realised they worked when played by projectM itself as a playlist. This is very good news! Means all adjustments in ConsumeBuffer and drawBackground are ALMOST done; its just a matter or understanding why when we select via the menu, some presets do not work. NICE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[Gustavo L Conte]

projectm_->initRenderToTexture();
 Resize(sceneRect().width(), sceneRect().height(), container_->devicePixelRatio());

dunno if this change anything, but I forgot to mention that: I've put on the end of the Init() function

PS: its also better to put like 15 seconds to test the playlist. THe presets seem to have kinda of a "initlization" mode and trigger the visualization after about five seconds when fed by the ConsumeBuffer

[Gustavo L Conte]

Considering what I posted before, about the way selectPreset(index) brakes some presets, making it impossible to preview via the interface, I've made this disgusting, ugly hack to enable preview. The problem is that the indexes are incremented in projectM, so I have to pick index-1 making impossible to get index = 0 which would be the first preset of the list LoLLL

void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
  }
#else
  if (projectm_) {
    if ( index <= 0 )
      return;
    Lock(false);
    projectm_->changePresetDuration(1);
    projectm_->selectPresetPosition(index-1);
    // Create a QTimer to call Lock(true) after a delay
        QTimer::singleShot(500, this, [this]() {
            Lock(true);
            projectm_->changePresetDuration(duration_);
        });
  }
#endif  // HAVE_PROJECTM4

}

pretty ugly, but its the proof of concept about what I've posted before. It works. Every single plugin works in the preview now, except the first one (yuck)

PS: the idea here is, instead of setting the preset directly, making projectM make a transition as if it was playing as a playlist queue of the main window. This way, it does not brake the preview, since they all work when projectM manages the transition.

[Gustavo L Conte]

Thats it! If we put those OpenGL commands in drawBackground,
change the static_cast to englobe both ( map.size / sizeof(size_t) ) /2 in ConsumeBuffer
with these changes in settings:

  s.smoothPresetDuration = 0;
  s.presetDuration = duration_;
  s.shuffleEnabled = false;

and my brand new ugly disgusting hack that now makes the preview work for the first element:

void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
  }
#else
  if (projectm_) {
    Lock(false);
    projectm_->changePresetDuration(1);
    if ( index <= 0 )
      projectm_->selectPresetPosition(projectm_->getPlaylistSize());
    else
      projectm_->selectPresetPosition(index-1);
    // Create a QTimer to call Lock(true) after a delay
        QTimer::singleShot(500, this, [this]() {
            Lock(true);
            projectm_->changePresetDuration(duration_);
        });
  }
#endif  // HAVE_PROJECTM4

}

I believe we have a working v3 integration with projectM! I executed the program for more than ten hours without segfaults or bugs.

[jonas]

@Gustavo-L-Conte said in Reintegrate projectM Visualizer:

if ( index <= 0 )
  projectm_->selectPresetPosition(projectm_->getPlaylistSize());
else
  projectm_->selectPresetPosition(index-1);

This looks wrong, since the index starts with zero, getPlaylistSize will be too high, needs to do - 1. Also, why are you using index-1 when the index is set? Doesn't IndexOfPreset return the correct index?

[Gustavo L Conte]

@jonas Its because i need to set the PREVIOUS preset, so that it elapses the transition, thats the only way I found not to bug some presets (that bug only on the preview); After fixing stuff, in the preview interface, some presets bug, i dunno why. But when they are playing normally as a playlist. when projectM is managing the transition, without us forcing with SelectPreset, this bug does not occur.

So I managed to do this ugly hack, that sets the previous preset, unlock, lets projectM do the transition, then "quickly" locks again after the timer

Read my previous posts, it was quite a journey. Thats the only thing missing I believe, to work with v3.

v2 is used on ubuntu and has some issues, i posted previously about that too.

The -1 would segfault, so I force the LAST preset to begin the first, when index = 0 is selected.

[Gustavo L Conte]

now my elegant BEAUTIFUL hack is gorgeous!

• we got ConsumeBuffer consuming
• we got drawBackground drawing
• we got playlist preview selector selecting and previewing
• we aint got no segfault, mon!

YES WE HAVE PROJECTM v3 (and v2 maybe) working

void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
  }
#else
  if (projectm_) {
    projectm_->selectPreset(index, true);
    projectm_->changePresetDuration(1);
    projectm_->setPresetLock(false);
    projectm_->selectPrevious(index);
  
    QTimer::singleShot(1500, this, [index,this]() {
      projectm_->setPresetLock(true);
      projectm_->changePresetDuration(duration_);
    }); 
     
  }
#endif  // HAVE_PROJECTM4

}

[jonas]

@Gustavo-L-Conte
short is the same as int16_t, 2 bytes because the consumed buffer is 16 bit, but size_t is 8 bytes, I'd like to understand why that is correct.
Another thing is that channels are hard-coded, so if the buffer has more then 2 channels, it will be wrong so we should pass channels to ConsumeBuffer

[Gustavo L Conte]

@jonas I really tried to understand that. I did put size_t because I saw somewhere that map.size type was size_t in gStreamer

The channels I tought it wouldnt be an issue, but thinking about it, it is.

###GstMapInfo

A structure containing the result of a map operation such as Memory.map. It contains the data and size.

struct GstMapInfo {
GstMemory* memory;
GstMapFlags flags;
ubyte* data;
size_t size;
size_t maxsize;
void*[4] userData;
void*[4] GstReserved;
}

its not the buffer, its the SIZE of the buffer, thats why its not 16 bit like the buffer itself!
Maybe thats why

[Gustavo L Conte]

This post is deleted!

[Gustavo L Conte]

Just found this, maybe useful

https://lwn.net/Articles/750152/

if (projectm_) {
  projectm_->setShuffleEnabled(false);
  projectm_->selectPreset(index, true);
  projectm_->changePresetDuration(1);
  projectm_->setPresetLock(false);
  projectm_->selectPrevious(index);

  QTimer::singleShot(1250, this, [index,this]() {
    projectm_->setPresetLock(true);
    projectm_->changePresetDuration(duration_);
    projectm_->setShuffleEnabled(true);
  });
}

Here's an improved version 0.1b to prevent altering shuffle globally in the settings. But don't forget to set smoothPresetDuration = 0 in the settings, this is required.

B I N G O
MilkDrop was heavily Windows-based, implemented with DirectX, Win32 APIs, and assembler. ProjectM did a good job of replicating the functionality in a cross-platform manner but one DirectX-specific piece remains: the shader code in the preset files. Some presets can contain GPU shader programs as mentioned previously. Because they were written for MilkDrop, they are in HLSL, a shader language for DirectX. Support for HLSL was provided in projectM by NVIDIA's Cg toolkit, but that has long been deprecated and is unsupported. Either manual or automatic conversion (possibly using something along the lines of HLSL2GLSL for Unity) needs to be added along with code to compile and upload the shaders. This would greatly increase performance and capabilities, enable the most advanced presets, and drop the dependency on an out-of-date and unsupported proprietary framework.

I saw on some presets some kind of shader language. The ones that bug!

[Gustavo L Conte]

in time, i just saw initializeGL with only BLEND command. All of those are required!!! Sorry if I did not express myself correctly. BLEND does the trick for the blank/black screens, but some visual bugs happens in lots of presets without the complete set.

glShadeModel(GL_SMOOTH);
glClearColor(0, 0, 0, 0);
glViewport(0, 0, width(), height());
glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glDrawBuffer(GL_BACK);
glReadBuffer(GL_BACK);
glEnable(GL_BLEND);

glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_LINE_SMOOTH);
glEnable(GL_POINT_SMOOTH);
glClearColor(0.0F, 0.0F, 0.0F, 0.0F);
glLineStipple(2, 0xAAAA);

ALSO, you need to run the commands always before renderFrame, at drawBackground method.

[Gustavo L Conte]

Now, here's another one to worry about:

A few days ago I installed those newest cream-of-cream set of plugins. Awesome by the way.
They seem all working fine. I spent days testing on random mode. I still want to compare one by one with the SDL app.

Now, problem is, when I go to the select visualization window, all the previews gets bugged if The first ones of the list (mostly called "Fast transition..") are selected. They don't seem to be normal plugins. Anyway, Thing is, if you select for preview any of these first ones, it bugs all other subsequent plugins. glClear color is not black anymore and they render all dumb. Only restarting Strawberry for normal behaviour again.
Gonna look at the milkdrop code, later, to compare the "normal" ones from these "Fast transition..." ones.

[jonas]

@Gustavo-L-Conte
Thanks for looking into it. It seems that projectm version 3 is buggy, I think we should try to get version 4 working instead, and drop support for version 3. See the discussion on https://github.com/orgs/projectM-visualizer/discussions/820

[Gustavo L Conte]

@jonas roger that, gonna go back focus v4
By the way I did compile latest git from v4 with his fixes, and tested with branch visualisations, did not work, same results. I'm curious about QOpenGLWindow results, with his fixes.

[Gustavo L Conte]

I had success starting to display stuff with v4 using QOpenGLWindow in a separate, smaller program. Soon I'll upload the code, I just need to adjust some things like resize and OpenGL context. I've used as ConsumeBuffer the pulseaudio capture code. I just need to adjust some stuff i think to make it better, but i got so excited about a positive display in v4 that i wanted to share ASAP

LoooooooooooooooooooooooooooL!

PS: still, several GL error 1282 in the majority of presets tested.

while some actually works

[Gustavo L Conte]

current status

the FPS on the video is very low due: * frameskipping at the desktop recorder

• fps was set t o 10 fps for testing purposes on projectm instance

[Gustavo L Conte]

I'm probably NOT the founder of the I.O.G.L.N.U.S.A.H.A.

Tthe International Open GL NEVER Ugonna SLEEP AGAIN ha ha ha Society.

Decided to go ahead and do the QOpenGLWindow on Strawberry.

Its worse than just the standalone program. Much more bugs. But the few that works are incredible. We are close. Here's what I've been messing around with my IogLnusaha association (I'm on Gnomes)

// Create and setup the QOpenGLWindow
  auto *openGLWindow_ = new VisualizationOpenGLWidget(projectm_visualization_);
  QSurfaceFormat format;
  format.setVersion(3, 3);  // Set OpenGL version to 3.3
  format.setProfile(QSurfaceFormat::CoreProfile);  // Use the core profile
  format.setDepthBufferSize(24);
  format.setStencilBufferSize(8);
  openGLWindow_->setFormat(format);
  openGLWindow_->resize(1280, 720); // Default size for the OpenGL window

  // Wrap the QOpenGLWindow in a QWidget container
  auto *glContainer = QWidget::createWindowContainer(openGLWindow_);
  glContainer->setFocusPolicy(Qt::TabFocus);
  glContainer->setSizePolicy(QSizePolicy::Expanding, QSizePolicy::Expanding);
  glContainer->setParent(this);
  glContainer->setVisible(true);
  ////////////setCentralWidget(glContainer);

  // Add the overlay as a child of the container and set its visibility
  ////////////////overlay_->setParent(glContainer);
  /////////////////overlay_->resize(320, 200); // Resize the overlay 
  //////////////////overlay_->setVisible(true);

  }

#ifndef VISUALIZATIONOPENGLWIDGET_H
#define VISUALIZATIONOPENGLWIDGET_H

#include "config.h"

#include <QOpenGLWindow>
#include <QOpenGLFunctions>

class ProjectMVisualization;

class VisualizationOpenGLWidget : public QOpenGLWindow, protected QOpenGLFunctions {
  Q_OBJECT

 public:
    explicit VisualizationOpenGLWidget(ProjectMVisualization* projectm_visualization, QWindow* parent = nullptr);
  //explicit VisualizationOpenGLWidget(ProjectMVisualization *projectm_visualization, QWidget *parent = nullptr, Qt::WindowFlags f = Qt::WindowFlags());
  //explicit OpenGLWidgetContainer(QOpenGLWindow* openGLWindow, QWidget* parent = nullptr);
  void initializeGL() override;

protected:
  void paintGL() override;
  void resizeGL(const int width, const int height) override;

 private:
  void Setup(const int width, const int height);

 private:
  ProjectMVisualization *projectm_visualization_;
};


#endif  // VISUALIZATIONOPENGLWIDGET_H

.cpp

#include "config.h"

#include <QPainter>

#include "core/logging.h"
#include "visualizationopenglwidget.h"
#include "projectmvisualization.h"

VisualizationOpenGLWidget::VisualizationOpenGLWidget(ProjectMVisualization *projectm_visualization, QWindow *parent)
  : QOpenGLWindow(NoPartialUpdate, parent),
    projectm_visualization_(projectm_visualization) {

    }

void VisualizationOpenGLWidget::initializeGL() {



  QOpenGLWindow::initializeGL();
  QOpenGLFunctions::initializeOpenGLFunctions();

    projectm_visualization_->Init();
  

}

void VisualizationOpenGLWidget::paintGL() {

  QPainter p(this);
  //////////resizeGL(width(), height());
  p.beginNativePainting();
  projectm_visualization_->RenderFrame(width(), height());
  p.endNativePainting();
  update();

  qLog(Debug) << __PRETTY_FUNCTION__ << glGetError();

}

void VisualizationOpenGLWidget::resizeGL(const int width, const int height) {

  
  Setup(width, height);
  projectm_visualization_->Resize(width, height);

}

void VisualizationOpenGLWidget::Setup(const int width, const int height) {

  glShadeModel(GL_SMOOTH);
  glClearColor(0, 0, 0, 0);
  glViewport(0, 0, width, height);
  glMatrixMode(GL_TEXTURE);
  glLoadIdentity();
  glMatrixMode(GL_PROJECTION);
  glLoadIdentity();
  glMatrixMode(GL_MODELVIEW);
  glLoadIdentity();
  glDrawBuffer(GL_BACK);
  glReadBuffer(GL_BACK);
  glEnable(GL_BLEND);

  glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
  glEnable(GL_LINE_SMOOTH);
  glEnable(GL_POINT_SMOOTH);
  glClearColor(0.0F, 0.0F, 0.0F, 0.0F);
  glLineStipple(2, 0xAAAA);

}

ConsumeBuffer:

void ProjectMVisualization::ConsumeBuffer(GstBuffer *buffer, const int pipeline_id, const QString &format) {

  Q_UNUSED(pipeline_id);
  Q_UNUSED(format);

  GstMapInfo map;
  gst_buffer_map(buffer, &map, GST_MAP_READ);

#ifdef HAVE_PROJECTM4
  if (projectm_instance_) {
    const unsigned int samples_per_channel = static_cast<unsigned int> (map.size / sizeof(size_t)) / 2;
    const int16_t *data = reinterpret_cast<int16_t*>(map.data);
    projectm_pcm_add_int16(projectm_instance_, data, samples_per_channel, PROJECTM_STEREO);
  }
#else
  if (projectm_) {
    const short samples_per_channel = static_cast<short>(map.size) / sizeof(short) / 2;
    const short *data = reinterpret_cast<short*>(map.data);
    projectm_->pcm()->addPCM16Data(data, samples_per_channel);
  }
#endif  // HAVE_PROJECTM4

  gst_buffer_unmap(buffer, &map);
  gst_buffer_unref(buffer);

}

InitprojectM

// Create projectM settings
#ifdef HAVE_PROJECTM4
Q_ASSERT(projectm_instance_ == nullptr);
Q_ASSERT(projectm_playlist_instance_ == nullptr);
projectm_instance_ = projectm_create();
projectm_set_preset_duration(projectm_instance_, duration_);
// Set initial window size
  projectm_set_window_size(projectm_instance_, 1280, 720);
  // Additional ProjectM setup
  projectm_set_mesh_size(projectm_instance_, 32, 24);
  projectm_set_fps(projectm_instance_, 60);
  projectm_set_aspect_correction(projectm_instance_, true);
  projectm_set_hard_cut_enabled(projectm_instance_, true);
  projectm_set_hard_cut_duration(projectm_instance_, 10);
  projectm_set_hard_cut_sensitivity(projectm_instance_, 1.0);
  projectm_set_beat_sensitivity(projectm_instance_, 0.5);
  projectm_set_soft_cut_duration(projectm_instance_, 10);
  
//projectm_set_window_size(projectm_instance_, 512, 512);
const char *texture_search_paths[] = { "/usr/local/share/projectM/textures" };
projectm_set_texture_search_paths(projectm_instance_, texture_search_paths, 1);
projectm_playlist_instance_ = projectm_playlist_create(projectm_instance_);
projectm_playlist_set_shuffle(projectm_playlist_instance_, false);

"patched" SetImmediatePreset (FOR TESTING, but works on v3)
void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
 /* projectm_playlist_play_previous(projectm_playlist_instance_, true);
    projectm_set_preset_duration(projectm_instance_, 1);
    projectm_set_preset_locked(projectm_instance_, false);

     QTimer::singleShot(1500, this, [index,this]() {
      projectm_set_preset_locked(projectm_instance_, true);
      projectm_set_preset_duration(projectm_instance_, duration_);
    }); */
  }
#else
  if (projectm_) {
    projectm_->selectPreset(index, true);
  }
#endif  // HAVE_PROJECTM4

}

It appears that when it wraps to a QWidget, the shader codes don't like and start to stop doing its things. ITs not a matter of context, but its like a corruption happens when you wrap.
I could manage to bind key_S to open the visualizations selector, when setting as central, because I could not figure out why the overlay and interface don't work

[Gustavo L Conte]

this version I posted works MUCH better giving nice results with the cream-of-cream presets,

[Gustavo L Conte]

void VisualizationOpenGLWidget::paintGL() {

  QPainter p(this);
  p.beginNativePainting();
  
  int w = width(); int h = height();
  this->resize(w*2,h*2);
  if (projectm_visualization_) {
    projectm_visualization_->Resize(w*2, h*2);
    projectm_visualization_->RenderFrame(w*2,h*2);
  }
  update();
  this->resize(w,h);
  if (projectm_visualization_) {
    projectm_visualization_->Resize(w, h);
    projectm_visualization_->RenderFrame(w,h);
  }
  update();

   p.endNativePainting();

  qLog(Debug) << __PRETTY_FUNCTION__ << glGetError();

}

Its disgusting, but it really makes a HUGE percentage of presets to display instead of black screen.
Ive done this without wraping the openglwindow into a widget;
when you resize the window, it displays, several plugins happens that way.. actually the nicer ones like this one: