Farm PCs locking up - *solved*

[kdm]

I have 3 farm PCs that are randomly locking up while working on one project. I have orchestral libraries spread among them (strings on one, brass on another, woodwinds/perc on another). I have Midi Over Lan running to all three from the host.

Any ideas how/why they would be locking up? I have to hard reset each to restart, and the last time my Vista 64 system with Cubase 4.5 hosting Play locked up, it corrupted the Cubase project (it would blue screen every time the first notes of this piece played back) and I had to rebuild it. I've contacted Alex at Musiclab, but am looking for other advice, troubleshooting suggestions.

I also get stuck notes and have had to activate "reset on stop", which used to cause other problems with MOL. That alone is making finishing this piece extraordinarily difficult, having to stop every 1-2 minutes to reset a stuck note, and avoid looping/cycle record, which is a major inconvenience.

This project isn't any more dense than others that so far have been fine. It also seems to happen more, or triggered by looping a section, though I can't be positive of that - I'm also returning to marker points on the fly etc.

Makes me think Nuendo may have some serious midi reset/note off bugs esp. in cycle mode.

Does this sound like a Midi Over Lan issue, or could it be an ethernet switch problem? I have a Netgear 8-port Gigabit switch in line here. The Netgear is pretty warm (wish I had a Sniffer to put on the network and just watch for garbage). I'll probably put a DLink switch back in later and just use the Netgear for an less used part of the LAN.

[kdm]

Just had the 3rd PC blue screen when trying to check system resource (fireface.exe/sys was listed in the bluescreen this time - the Vista 64 system bluescreen an hour ago, twice, and the dump request listed molcp3 (Midi over lan driver).

What the crap is going on???? All 3 farm PCs crash two days in a row? The host hasn't crashed yet, so this has to be a networking problem on the receiving end. Heck if I'm dinking around with this anymore - I'm putting a new switch in line and see if that cures it, but dang, I'm having a hard time figuring out how a PC would crash with no significant network data happening.

I think one of the two older PCs has a 100Mb NIC. I'm thinking of putting a 1Gb NIC in it to see if that helps, along with a replacement switch, though I don't have enough ports so the Netgear has to stay in the network.

[TAFKAT]

Dedric,

Send me the dumpfile, and I'll analyse it..

I should be able to pinpoint whats bring the system down for you.. :-)

[kdm]

Thanks Vin that would be a huge help - where would I find the bluescreen dump file? (Vista 64)

[kdm]

Found it - memory.dmp - it's 351M so I'll upload it to my ftp and pm you a link. I can also include the WindowsUpdate and molcp3 (Midi Over Lan's boot report I think) text files from a subsequent reboot if that will help.

[Jcschild]

VSL is about to release their updated sample player that will allow for 3rd party VSTi in both 32/64 bit flavor at the same time and with 3rd party plugs.....

this may fix alot of issues.... and yes its lan based or interface based or both


Scott

[kdm]

Yes, I have my eye on VEPro for just this situation, assuming it works with Play (otherwise I'm waiting for Kontakt 3.5......), and assuming the problem is MOL itself, and not a Nuendo midi problem.

The more immediate problem though is that I have work to do today, tomorrow, Monday, etc, and don't have enough midi interfaces to replace MOL completely, though I am considering doing just that, as costly as it will be.

I can't wait for the next new app, or the next bug-fix release to get a system that works. I'm tired of our apps, and even our computers being one step behind the work requirements in stability and reliability. I setup farm PCs and intend to continue that config specifically to prevent overloading any one machine. So much for that theory...

...but I digress.

Each of my farm PCs has a different config - one uses Cubase 4.5/64-bit hosting Play/64 on Vista64, the other two run Kontakt on XP32.

Either midi over lan or Nuendo has to be doing something outside of ethernet spec to crash 3 PCs with lan traffic alone.

I know quite a few composers are running Midi over lan successfully, but maybe they are just are using their farm systems differently, or maybe it's just some unique quirk in this project, arrangement, or combination of midi data - it's really odd that it would blue screen the Vista machine at the same point in the piece, but only after locking up one of the other PCs, then the Vista machine, first.

This piece is a fast moving action/chase cue, but even still, it's not like I have 16th note runs at 160bpm in every section simultaneously, and I would just about expect 1G lan to be able to handle that, at least in a torture test.

I might post this over on VI to see if any other guys with multi-PC setups have run into this.

[paulwr]

MOL is such simple software written in a way that just should not interfere with anything else............

Makes me wonder if you might have a virus going on.... do you use any of the computers online?

You keep c drive images of all the machines, right? I'd be restoring them and see if that handles it.

-Paul

[kdm]

I have another PC for online use - none of the farm PCs goes online. I also ran a virus scan on several systems last week just to be sure they were still clean with the conficker worm working its' way around.

MOL becomes suspect as it was listed as the crashing app when the Vista system BSOD'd. The potential for a virus did come to mind, but if it is, it isn't showing up in scans.

[paulwr]

Wild problem. I've never had all my slaves go south at once like that, and along with the DAW. Yikes. Are you able to restore all the c drives from a disk image? If working well before, should work well again.

Good luck,
-Paul

[kdm]

Hi Paul - the host didn't crash - just the slaves, and only 2 at a time. I am now thinking the Vista system is BSOD'ing due to a bug in Cubase 4.5/64 that is triggered when it locks up (i.e. the project is corrupted - typical Cubase problem...sigh). The lockup problem I am hoping is Midi Over Lan. I had an older version of MOL on the other two farms, and the latest on the host and the Vista system. I updated the two older MOL farms hoping it's a network issue within MOL 3.0 that was fixed by 3.3.

[paulwr]

Check in with the guys at MusicLabs. I think he's back from vacation and might have a suggestion. Still, rare the basic source would be mol. Sometimes it can be something from left field. I had a basic MidiSport several years ago that never messed up on just one computer, but screwed Cubase sx 3 up royally once the slaves were added, and turned out to be a known issue. Midi was sliding as a block way out of time. The guys at VisionDAW who sold me computers are are extremely capable with large music computer networks told me there was now way it could be mol, and spent 3 hours on the phone with me until they solved it. Great guys.

Hope you have luck soon on that.
-Paul

[kdm]

Haven't had a lockup (yet) with any of the farm PCs since replacing Cubase 4.5 on the Vista64 system with Bidule.

The Vista system does infrequently get caught in a short midi loop, ending with a stuck note, but resetting that Play instance's audio engine cures it - annoying, but infrequent, and not the bluescreen/crash that Cubase 4.5 would cause. That's only happened twice, but I've also stopped using cycle record in Nuendo since that seemed to cause stuck notes, and lockups more often than not. Wonderful to be giving up features to avoid problems that developers should fix.... grrr.

However, stuck notes continue at a workflow-hindering pace. Anyone use any other midi over lan solution beside MOL?

I'm thinking of switching if ipMidi or something else will work better. VSL Pro isn't out yet to demo, that I know of, and there is no guarantee it will be a solid solution for hosting Kontakt and Play just yet.

This is really frustrating. So far no fix from Musiclab, but there may not be much he can do to solve it unless he can recreate it there.

[Daryl]

I'm thinking of switching if ipMidi or something else will work better. VSL Pro isn't out yet to demo, that I know of, and there is no guarantee it will be a solid solution for hosting Kontakt and Play just yet.

I have no suggestions for your MIDI problem, but some time over the next few weeks I suggest that you check with EW and NI, in order to confirm they are intending to test their software with VE Pro, so that it becomes a "supported host". EW is famous for not testing on some hosts and then refusing to help solve problems. :icon_mad:

D

[kdm]

Changed subject - it seems this may be multiple problems:

1 - Crashing Vista system seems fine since replacing Cubase 4.5 with Bidule. Not sure why or how Cubase could cause lockups within the MidiOverLan network, but that seems to have stopped.

2 - Almost positive "Reset on stop" in Nuendo over a Midi Over Lan network can cause blue screens on some systems. No idea why. I seem to recall it being an issue with Chaingang at one point.

3 - Kontakt 2/3 does seem to have issues with stuck notes. No solution, but I've seen this reported on other forums with Logic, DP, Reaper and Cubase. Just had a stuck note in Kontakt 3.0.2 on the host (Nuendo) system as well, so it doesn't seem isolated to midi over lan recipients.

4 - Stuck notes continue, but mainly on the one Kontakt 3.0 PC (was on K2.1.x). Could be Kontakt.

5 - Had a stuck note while in the Key editor soloing a midi part going to a completely different PC. Not sure why Nuendo would, or could send midi data to any other destination when in solo mode. Wondering now if this is an extension of the solo bug, or just a coincidence.

[Beat Productions]

I'm using IPMidi now and I haven't had any problems. I used to use the original MOL and I can't remember why I chose IPMidi over upgrading MOL, but I'm happy.

[kdm]

Beat Productions - any issues with having "reset on stop" enabled in Cubendo with ipMidi slaves? It can crash MOL slave PCs, so I never enable it anymore (actually blue screens one PC).

[Beat Productions]

Beat Productions - any issues with having "reset on stop" enabled in Cubendo with ipMidi slaves? It can crash MOL slave PCs, so I never enable it anymore (actually blue screens one PC).


I never use "reset on stop" anymore, I have a program callred "Midi Translator" by "Bome" that I use for various midi functions/translations but one thing I use it for is to route a qwerty keyboard key (in my case the / key) to send a panic (all notes off) command to my main DAW (Cubase 5) and remote Daw (Cubase Studio 5, Reaper and/or Bidule) via a qwerty to midi translation.

[paulwr]

bidule for Play on a slave is on the right track. Vision DAW sets up their composer system slaves with it, but I seem to remember them using a particular version.....wish I knew more.

Kontakt2........ update to the latest for K2.2.4 -I updated from a much older version a few months ago, and it is worthy.

Other than that, I just don't know. All my slaves and DAW are 64 bit except my Giga machine. I'm not suffering any of these problems. Now, I am still on Cubase 3.1 Vision DAW guys, who serve many many composers both big and small in LA, say about 40% of the Cubase/PC users are still on v 3.1 as well.

Good luck,

-Paul

[kdm]

Finally figured all of this out:

1 - Lockups were caused by Cubase 4.5/64 bit preview - no idea why or how, but replacing it with Bidule cured the lockups on all three machines. That's one for the books....

2 - Stuck notes - I had single standalone instance of Kontakt 2/3 (tried 2.1, 2.2.4, and 3.0) on the one slave but apparently it had reached the 2G limit, being a 32-bit app. Splitting it to 4 instances in Bidule with the /3G switch made better use of ram (I assume). No more stuck notes. Again, Bidule saves the day. Great app for sure.

Fyi - I'm using the latest versions of Bidule - 32-bit on the system that had stuck notes, and the 64-bit version on a Vista system with 8G ram hosting string libraries. No problems with either version so far.

Thanks for the assistance all!

[TAFKAT]

Finally figured all of this out:

1 - Lockups were caused by Cubase 4.5/64 bit preview - no idea why or how, but replacing it with Bidule cured the lockups on all three machines. That's one for the books....

:icon_eek3:

I would never have expected that... :eusa_whistle:

[kdm]

:icon_eek3:

I would never have expected that... :eusa_whistle:

:icon_lol:

Technically it doesn't make any sense. Non-technically speaking... maybe I shouldn't be surprised....:eusa_whistle:

Wonder if Cubase 5/64 would do the same thing.... yikes.

[paulwr]

Hey, get those K2 slaves rolling on 64bit o/s! You'll be glad you did. Also, there is a way to get two instances of K2 on a slave, each able to access 3.6 GB of RAM if you have 8 GB on the machine. It will be awhile before I get to that here since I'm in the process of transforming a room to a composing room after having just relocated. I'll post how it goes once I do it. Supposed to be rock solid.

-Paul

[Beat Productions]

BTW There's a 64 bit beta of Kontakt out now, I don't have Kontakt yet so I can't comment.