file formats WK 7600 - Structure of file formats ( i.e. AC7, TN7, RM7, AL7 )

[markonis]

Hello guys,

 

I'm a software developer and I'd like to build a few applications that will extend the functionality of this fantastic keyboard even more.

For this I need to know the structure of the files mentioned in the title - that is, how the data in these files are physically stored and structured.

I already contacted the CASIO company and I'm waiting for their response, however I'm trying to seek the answer also among you in case CASIO will not want to share it with me.

Could you, please, point me to a right person, website, ... where can I get this info.

 

Thanks very much.

Marek

[Jokeyman123]

This is a tremendous idea, one that some of us have discussed-especially a software front-end that could translate between song or rhythm formats that are not now capable of doing that....however I don't think Casio will allow you to disassemble their proprietary code for any of the software. This website has some good development tools specifically for music hardware and software which might help along with the various midi implementation charts for specific Casios...

 

http://ctrlr.org/

 

This development website requires a good command of sysex and other midi programming but enables development of software programs for any of 3 platforms-Linux, Windows or Apple. What specifically are you thinking of developing? There are a few here that probably have the programming chops to help you, moreso than I can.

[markonis]

Hi Jokeyman,

 

it is exactly as you've said - CASIO responded that the file formats is internal information not available to the public.

I'm planning to develop a front-end for WK 7600 which would allow me to build the following functionality:

1) management of registration banks:

       1.1 see reg. bank information - which rhythm, tones, settings is used for the particular reg. bank / slot

       1.2 edit the slot info - play with different rhythms, tones, settings

       1.3 merge multiple reg. banks - so that the user can save slots from different reg. banks alongside with a current loaded reg. bank

       1.3 save the reg. banks info back to a file 

       1.4 re-upload the reg. banks to a keyboard

 

2) management of user rhythms:

     2.1 create rhythm parts from a specific midi file ( for ex. copy a part of midi file and set it as INTRO, FILL-IN, .. ),

     2.2 edit rhythm in real-time, i.e. re-assign tones / channels,  mute channels, set volume and other available settings

     2.3 save the user rhythms info back to a file 

     2.4 re-upload the user rhythms to a keyboard

 

3. management of user tones in real-time through front-end:

    3.1 better use of sliders, mod wheel, ... for a manipulation of a sound/tone

    3.2 modification of tone through keyboard is not so intuitive - using a front-end would be a quicker

    3.3 save the user tones info back to a file 

    3.4 re-upload the user tones to a keyboard

 

4. manage sets of reg. banks, user rhythms, user tones

    4.1 inter-connect the banks with rhythms and tones so that a user can have categories /sets ready for upload at once

           ( let's say dance category would upload specific reg. banks, related user rhythms, user tones suited for a dance songs, 

    4.2 re-upload specific user set to a keyboard in one step

 

The most of the above functionality should work both ways - change in application should be immediately noticeable in keyboard

and change in a keyboard setting should be reflected in the app.

 

In addition to the above my idea is to build a website for this community of users that would allow them to exchange reg. banks, rhythms, tones, user sets

since this is missing most in my opinion.

 

All of this, however, strongly depends on the knowledge of the relevant file formats.

 

MarekP

[shiihs]

@markonis Did you make any progress with this endeavour? 

 

[markonis]

Hello shiihs,

 

Sorry for the long delay but I was unable to log in to my account.

The website that Jokeyman123 pointed me to has brought me to a realization that I don't need to parse the files exported by CASIO keyboard. 

I found out that there's an official documentation for a WK 7600 describing all MIDI events that the keyboard is capable of sending/receiving.

Via SysEx messages I'll be able to get to the most of the data needed for my project.

 

So the answer to your question is that now I'm able to build the application though I'll need to go really low level that means quite a bit of tedious work in order get all the details.

Now I'm in the phase of building the  framework ( basic engine ) upon which I'll be able to build all the modules.

 

MarekP

[Jokeyman123]

Wow, this is fantastic if you can do it. I do not have the sysex "chops" for building this type of application although I've wanted to do so but I would definitely like to see how you manage. I also have a CTK6200 in addition to my Privias. I'm sure there are many users here, including me who will be glad to be "beta testers" for this software at any stage of the development. Best of luck.

[markonis]

Hi Jokeyman,

The SysEx route in addition to all the midi events processing staff is quite challenging ( there's a lot to study ) but after few successful tests I'm now pretty sure I can do it. I was able to bulk export parameters for one specific CASIO preset tone from the keyboard using sysex's HBR ( Handshake Bulk Parameter Set Request ) transfer method. So the bulk data transfer is functional. However, I also need to send / receive individual parameters in order for the app to work correctly. 

 

But currently I can't manage to export any specific parameter using IPR ( Individual Parameter Receive ) transfer method - I'm still getting REJECTED error message from a keyboard. I will probably need to see some examples of such SysEx message - maybe some users in this forum were successful with this.

I feel that I'm not specifying parameters "blk" / "idx" / "len" correctly when I'm building this type of a sysex message.

 

The other issue which I need to solve is the markers for all the sections ( Intro, Variation 1, 2, ... ) in a CASIO rhythm.

I will also need an example of a midi file with these markers specified.
Until now I didn't find a list of CASIO rhythm section markers ( only for YAMAHA styles ).

From what I have learned I know that the app will be able to cover these models ( CTK-6200 / CTK-6300 / CTK-7200 / CTK-7300 / WK-6600 / WK-7600 ).

The differences between these models are basically: 1) number of keys ( i.e. octaves ), and 2) number of presets ( built-in tones / rhythms , user tones / rhythms, ... ).

 

One issue I know of could be with keyboard models designed and shipped to Indian market. The mapping and internal numbers of built-in tones and rhythms will probably differ from those shipped to US/EU markets. In reality it could mean that when "Indian" user will choose let's say Grand Piano the keyboard will play some other instrument.

 

The above paragraphs show that I will surely need some cooperation from users ( in the future ).

That's why I was also thinking about getting few of the users involved in the development process as beta testers.

I already see that the project is quite large and that each module will need to be developed separately with exception of the underlying framework that needs to interlink all these modules internally.

 

As you can see the development will consume a lot of my private time ( I'm a full-time SW developer ) so I'll probably need to set some price for the app as a compensation. The beta testers will surely have some advantages as the testing will also take some time on their side.

One more thing - as I'm a technical person ( not an artist ) the UI ( user interface ) of the app doesn't necessarily will have to be perfect. 

This is where a user-designed layouts can be helpful for me.

My idea is to keep the look of the app as close to a keyboard one so that users can easily refer to documentation and/or keyboard LCD screen.

 

So, when I'll have something working to show I will make the application available - we can then discuss details of a cooperation.

Since the project is opened one and I'm willing to incorporate any good ideas for a functionality of the app any suggestions and recommendations are welcome.

 

MarekP

[shiihs]

That's good news. If possible please consider using a cross-platform framework so linux users like myself can benefit from it as well.

 

In the mean time, I've started working on a parser/writer for ac7 files. My current plan, should I succeed to bring it to completion, is to make it available as open source with GPLv3 license on github so perhaps others can use it as a starting point to build e.g a style composer. In the (likely?) absence of "others" I might give it a stab as well, but from where I stand now that would be far in the future.

 

I don't really understand why Casio is so protective of this information - from what I have seen so far, there's nothing in there worthy of special protection (in the sense that it doesn't contain unique features that don't exist in more or less the same form with other makers of keyboards). 

 

[Chandler Holloway]

5 hours ago, shiihs said:

In the mean time, I've started working on a parser/writer for ac7 files. My current plan, should I succeed to bring it to completion, is to make it available as open source with GPLv3 license on github so perhaps others can use it as a starting point to build e.g a style composer.

 

This sounds like a very intriguing project. Several of us here on the boards have requested an official .AC7 editor to replace the now decade old .CKF Rhythm editor from the IDES 4.0 software suite. I wrote a reference manual that addresses some of the details of the old .CKF format and how to use the older software, you can find it here for some further reading. Most .AC7 keyboards are backwards compatible with the .CKF format and will automatically convert a .CKF file to the .AC7 format upon uploading it to the keyboard, so the old IDES editor does still address the problem somewhat, but the .CKF format is limited when compared to the .AC7 format in two very important ways: number of usable Rhythm parts (.CKF can only use five parts on MIDI channels 10-14, whereas .AC7 Rhythms can include eight parts on channels 9-16) and number of usable variations (.AC7 files can contain pattern data for four separate variations and associated fills, whereas .CKF files only have data for two). An .AC7 software utility would address these problems and be greatly appreciated by the community here, so I thank you for your efforts in this regard.

 

I don’t know if this feature would be within the purview of your .AC7 parser/writer, but one (hopefully simple) utility that I know would immediately have an impact on very many Casio users is an .AC7 “Rhythm renamer” for Windows and Mac. Obviously the Rhythm file itself can be renamed on your computer for archival purposes, but the file label is completely distinct from the display name that actually shows up when the Rhythm is loaded onto a keyboard and selected in the Rhythm list. None of Casio’s official software that can handle the transfer of .AC7 Rhythms is capable of changing the display name, however, so the only option for renaming at the moment is to own a Casio keyboard with an onboard Rhythm editor and do it on the hardware itself.

 

The reason why custom display names are so important is that they enable a very clever workaround for an issue that many Casio users have; the inability to name User Registrations on most Casio keyboards (only the high-end MZ-X models have this feature). On most Casio keyboards with User Registrations, a registration is only identified with a number, such as “Reg 1-1” to signify the first registration in the first registration bank, or “Reg 2-3” to signify the third registration in the second bank. If you’re using a lot of registrations, it can be hard to remember all of them with just these numeric identifiers, so Casio users have been asking for a way to name registrations for a while.

 

While only the MZ-X series can properly name Registrations, many Casio keyboards in the current line-up can import custom User Rhythms, which can be named. So the workaround for this issue is to load up a User Rhythm with a custom name of your choosing and then save this Rhythm file as a Registration. The Registration remembers the Rhythm file you had selected when you recall it, so users can simply use the custom Rhythm name as a makeshift registration label to help keep things organized. However, given the present inability to modify the display name of a User Rhythm unless you have a keyboard that includes an onboard Rhythm editor, this workaround isn’t achievable for a lot of Casio users who would still find it useful. 

 

Many Casio keyboards can import custom User Rhythms created elsewhere, but considerably fewer models actually have an onboard Rhythm editor capable of display name editing. Even if such boards could rename rhythms (but not edit them), these owners of “Import but not Edit” Casio keyboards would only be able to feasibly work with the preset Rhythms anyway, and even then there would be no way for them to copy them over into the User Rhythm slots for the purpose of exporting them, editing the display name on one’s computer, and then re-importing them to act as makeshift Registration labels. Thus, there would need to be a repository of these Rhythms archived somewhere externally (likely here on the forums) for owners of the keyboard to use in order to achieve this workaround.

 

Most of the time, Casio’s Rhythm files are not truly unique to each keyboard in the line-up such that each keyboard has a distinct set of however many rhythms. Rather, the Rhythm styles are often shared among keyboards of the same “generation”, and more advanced models of that “generation” will have additional bonus rhythms in addition to the standard set of rhythms found on the lower end models. For example, the recently released CT-X models and the CDP-S350 are all part of the same generation of Rhythm styles (since they all use the new AiX sound source and the styles were written to take advantage of that). The CT-X700, X800 and CDP-S350 are all “import but not edit” boards with regards to rhythms, and they all share the same set of 200 Rhythms. The CT-X3000 and X5000 have onboard Rhythm editors as well as additional Rhythms not present on the lower end models.

 

As a result of my work for Casio, I have regular access to many different models of their keyboards and took advantage of this to release an “expansion pack” of Rhythms for CT-X700/800 users consisting entirely of the exclusive CT-X3000 and X5000 Rhythms so that CT-X700 and X800 owners could have access to these exclusive Rhythms on their keyboards. If I knew that there was a way to modify the display name of a User Rhythm for use in the aforementioned Registration label workaround, I would happily take up the task of archiving all the preset .AC7 format Rhythms of the Casio keyboards I could get my hands on. For starters, the CT-X series, then the most recent generation of WK models, and hopefully eventually the MZ-X series. It is a tedious task to export and rename all of these Rhythms on the hardware itself, but with a software solution I would be able to get through it all much more efficiently (and then end users would be able to rename them again to make use of the registration label workaround).

 

All of the “import but not edit” keyboards that would benefit from the existence of such an archive are “two variation boards”. Only the CT-X3000/X5000 and the MZ-X models are “4 variation boards”, and they have onboard editors so they don’t have a use for the archive in the first place. With that in mind, if I was to archive Rhythms from a 4-variation board for use on a 2-variation “import but not edit” model, I would archive each 4 variation Rhythm in two parts; a “1-2” file which contains the first and second variations of the source 4-variation style, and a “3-4” file which contains the third and fourth variations of the source file. Thus, 2-variation “import but not edit” models would have access to all four variations in addition to the ability to use the registration label workaround. Thanks to @vbdx66 for recommending this back when I released my original expansion pack.

 

Sorry for the massive wall of text, but I hope that this has helped you understand just how immediately useful a Rhythm renamer program would be to tons of Casio users, not just now, but many years into the future. You would be doing us all a great service, and I’m happy to pitch in with the archival process to make the most out of it. Please consider releasing a standalone Rhythm renamer with a friendly user interface to help make this possible! You’d be a legend here forever. @markonis should feel free to help out as well, any assistance would go a long way.

 

I did a little bit of preliminary investigation into the .AC7 files myself (and by that I mean I opened them in notepad because I have no programming skill whatsoever), and I found that the display name is actually stored in plain text near the beginning of the Rhythm file. I tried just replacing it with a custom name by typing it in through Notepad to see if that would work, with the same number of text characters and everything to make sure I didn’t screw up the file too much. Unfortunately, this must have messed with the file format in unintended ways as CT-X Data Manager gave me a generic “File Format” error when I tried to import the modified .AC7 file into my CT-X700 afterwards. It appears that a more elegant solution will be needed to properly edit the display name, which is where I’m hoping you guys can help out. See the attached screenshot to get an idea of what I mean.

 

 

[shiihs]

53 minutes ago, Chandler Holloway said:

Unfortunately, this must have messed with the file format in unintended ways as CT-X Data Manager gave me a generic “File Format” error when I tried to import the modified .AC7 file into my CT-X700 afterwards. 

 

@Chandler HollowayI expect your approach to changing the name using an editor is basically valid (provided you don't change the name length), but notepad is the wrong kind of editor for this type of editing as it cannot handle binary files. So far in my (admittedly very early) investigations I haven't seen any checksums or other tricky things that might cause your approach to fail, so my guess is it should suffice to redo your editing with a file editor that supports editing binary files. Many free ones exist, e.g. see https://www.poftut.com/best-free-hex-editors-windows/ or https://www.slant.co/topics/1208/~best-hex-editors-for-mac 

 

@markonis to avoid reinventing the wheel and spending tons of effort on ui stuff, have you see this? https://github.com/eclab/edisyn/ - maybe interesting (or not - i'll leave that up to you )

 

 

 

[Chandler Holloway]

@shiihs I used a hex editor (XVI32) to edit the display name and it loaded in just fine! You are correct, though, in that any change to the number of characters in the original Rhythm file name will produce a file format error when you try to swap it in.

 

Thank you for your advice. I think a standalone renamer program with a GUI would still be most convenient and certainly more approachable for our less technically inclined users, but at least now I know it's possible. I'll have to document this in a dedicated forum post soon.

[shiihs]

5 minutes ago, Chandler Holloway said:

@shiihs I used a hex editor (XVI32) to edit the display name and it loaded in just fine! You are correct, though, in that any change to the number of characters in the original Rhythm file name will produce a file format error when you try to swap it in.

 

 

@Chandler Holloway Changing the length of the name is tricky because the ac7 file format embeds offsets to the start of each of the different section types in the binary data, and these offsets become invalid if you change the name length. Usually before a string (like the name of the rhythm), also the string length is embedded, so that needs to remain in sync as well.

 

Anyway, you can always use "space" characters to make too long names appear shorter, but making short names longer will take a smarter tool that actually understands the contents of the files. 

 

Your suggestion for making such renamer (which then correctly handles making longer names etc) therefore sounds like a good first application of a parser. 

Don't get your hopes up too high yet. It'll still take considerable work to get the parser in good shape (or any shape, really - I can only parse a first part of the files so far). *Sigh* Things would be a ton easier if we had extensive reference documentation on how the files are structured, but in the absence of such docs, there's a lot that can be done - it just takes (a lot of) time, motivation, dedication, some luck and (most importantly) a big dictionary filled with curse words 

 

[markonis]

@Chandler Holloway Thanks for the perfect guide regarding the structure of CKF files and applying markers to MIDI files. I was searching for this information for quite a long time.

 

@shiihs I was looking at the "edisyn" framework but it's not what I need for these reasons ( the following is not a critique just a summarization of facts ) :

  1) it seems to me that the framework tries to be universal tool so it only utilizes program changes (PC=tones) and controller changes (CC=effects) for GM 1 specification so that broad range of keyboards from different manufacturers can use it ( though the framework recognizes only CASIO CZ series - as per user guide ),
  2) maybe I could incorporate other CASIO models' ( via specific sysex messages and PCs / CCs ) using the framework but I'd need to learn the specific language/syntax to do the same what I'm doing in my project right now,

 

3) the most important:   the functionality of the framework looks like just another VST-like tool/app for editing tones but I'm planning to build something more specific, broader in scope and targeted primarily for CASIO keyboards ( see more details below )

 

@shiihs I'll probably disappoint you since I'm a whole-life Windows developer so I'm not able ( at least now ) to build a cross-platform framework. However, I'm planning to build 2 versions of the app: desktop and mobile ( for tablets ) so with a little investment also Mac users could benefit from my app ( is this feasible for you as an end user? ).

 

@shiihs Your idea to create AC7 parser looks interesting to me.

 

Based on my current analysis I have the following possibilities regarding access to internal CASIO data ( there are more but for now I'm focusing just on these ones):
 

1) import / export of user tones                                 - YES, as per single tone                              ( built-in tones - NO directly, YES indirectly - see below )
2) import / export of user DSP effects                      - YES as per single DSP effect                    ( built-in DSP - NO directly YES indirectly - see below )

3) import / export of user registration banks           - YES, all in one block  ( need to verify )    ( built-in RB - NO directly YES indirectly - see below )

4) import / export of user rhythms                             - YES as per single rhythm                          ( built-in rhythms - NO directly YES indirectly - see below )

5) import / export parameters of points 1) - 4) + mixer parameters
6) import / export of all data ( = AC7 file )                - export YES, all in one block, import YES only in case of being able to parse and identify each byte of data in the block

 

All of the above enables me to build an application with modules ( i. e. editors ) for each of the above points.

In addition to user data, I'm able also to export built-in keyboards data but it's quite a tedious process - I need to manually save each built-in user data ( for ex. tone ) as user data ( for ex. user tone ) and then export it.
All the parameters for user tones and DSP effects can be exported per item ( for ex. tone ) but reg. banks as well as all user data ( = AL7 file ) can be exported only as one continuous block of data which makes parsing it a lot more complicated.

Since I'm currently not able to parse the user rhythm data ( I don't know the internal structure of the data, for ex. where the rhythm sections start ) the user rhythm parser of AC7 file ( or at least CKF file ) would be of great help for me. 

Regarding user rhythms, at this stage of analysis I'm planning to build an easy MIDI file parser which should enable user to set markers in the imported midi file and copy & paste parts of midi file into separate rhythm sections + modify channels + apply effects to the tones so that it complies with CKF / AC7 file ( of course I'm not going to create another Anvil Studio - for complicated adjustments there are many good applications  ).
Such a modified midi file should then be processed via IDES application into proper CKF / AC7 file and imported into keyboard as  a user rhythm.

 

So it seems to me that both me and you, shiihs, are trying to handle the user rhythms but each of us from a different side.

@Chandler Holloway Based on the above points I should be able to at least change names of reg. banks and user rhythms so that I can create kind of a map between these two user data. I think that this is a functionality that you were asking for in your contribution, am I right?
As a working title I named this module in my app as a Registration Bank Editor.

 

MarekP

 

[markonis]

@Chandler Holloway All the above applies to models  WK xxxx / CTK xxxx since structure of sysex messages and internal allocation in keyboards data in memory for other models differ. In order to make the app usable by other models I would need to have the models at my disposal to test them ( the good thing is that technical specification of MIDI implementation for the other models is available online ).
From what I saw in specs for different models I personally think that the internal data could be very similar for different models but that would need to be verified and tested.
At this stage I need to build a proof of concept in the form of an app for WK 7600 which I have at home.

After that I can think how to make the framework universal and workable also for other models.

 

MarekP

[- T -]

Marek

 

Since, by your own statements, your scheme will rely heavily upon SYSEX message transmissions, you may need to take into account that the currently accepted moniker of "Fully Class Compliant" does not necessarily mean that "generic" USB-MIDI is fully compliant with the overall MIDI Specification.  It just means that devices and/or processes,  labeled as such, are fully compliant with the "generic" USB-MIDI spec.  Generic USB-MIDI, as implemented in Windows (since XP Service Pack 2) and contemporary versions of MAC-OS and iOS, and most contemporary keyboards and sound modules, do not support the X-ON/X-OFF flow control of the old RS-232C MIDI-DIN protocol to prevent receive buffer overflow during large file transfers and dense SYSEX message transmissions.  This is the reason that, for a time, both Roland and Yamaha required proprietary USB-MIDI drivers for their high-end keyboards (Motif/Fantom/FA/etc) that offered "Close Integration" with their respective DAWS (Sonar/Cubase), which relied heavily upon dense SYSEX message transmissions, but such flow control schemes only work when both ends, the software and the device (keyboard/sound module) support that flow control.  As stated above, current MOTL and entry-level keyboards do not support any kind of flow control on their USB-MIDI "TO HOST" ports.  Roland and Yamaha finally incorporated menu driven switches on their high-end models for the selection of proprietary or generic USB-MIDI drivers.  This phenomenon can be observed in the Casio CTK/WK-7XXX models, with their 100 User Tone and 100 User Rhythm memories.  A bulk transfer of 100 of either, with the Data Manager software, usually results in some 14 or 15 or 16 items missing, that were just lost during receive buffer overflows, and with no error message being generated.  At least my old WK-3800 lets me know that USB-MIDI buffer overflow has occurred by choking, locking up, and dumping just about eveything that was resident in User Memory.

 

Regards,

 

- T -

 

 

 

[markonis]

Thank you very much for this valuable information. I'll take it into account and test it properly.

[shiihs]

@markonis I'm not sure if this is the same thing that  @- T -  alluded to, but it is my experience in interacting with other synthesizers that sending/receiving very large sysex over USB can be tricky due to timing issues. I suffered specifically from the following problem: if one sends data to a hw synth too fast, the synth is not guaranteed to be able to process it in time and data randomly gets overwritten, potentially leaving the synth in a weird state. A second problem I suffered from was failing to limit the midi datavalues to range 0-127 in all cases. This too caused my synth to do really weird stuff by starting to interpret datavalues as commands. So be sure to clip the datavalues appropriately at all times (Just trying to help you avoid a few sleepless nights )

 

The solution for the timing problems when sending sysex to a synth normally is to "chunk" sysex messages (i.e. divide it into smaller blocks and insert a delay between sending the successive blocks). None of the available midi libraries that I'm aware of, however, seem to directly expose such chunking of sysex messages with delays in between out of the box (although popular low-level midi applications like midi-ox (windows) and sysex librarian (mac), and amidi (linux) all implement this technique).

 

I'm not certain if the CASIO sysex messages would be large enough to cause such problems - I haven't attempted making any dumps yet - but it may be something to keep an eye on from the start. Sysex of a few KB never caused me any issues (but even that may depend on the speed of the computer being used). I was trying to send sysex messages of several megabytes at the time. (I was reminded of this problem when @- T - spoke of missing data when using datamanager, which may in fact be the exact same problem I experienced).

 

Anyway, best of luck to everyone trying to further unlock the full potential of these machines.

 

[- T -]

4 hours ago, shiihs said:

 I'm not sure if this is the same thing that  @- T -  alluded to, but it is my experience in interacting with other synthesizers that sending/receiving very large sysex over USB can be tricky due to timing issues. I suffered specifically from the following problem: if one sends data to a hw synth too fast, the synth is not guaranteed to be able to process it in time and data randomly gets overwritten, potentially leaving the synth in a weird state.

 

shiihs:

 

Yes!  You have just defined "buffer overflow" as it applies to MIDI communications.  This phenomenon has been well known since the mid-1980's, but with the X-ON/X-OFF flow control of the RS-232C MIDI-DIN protocol, it was never much of a problem until USB-MIDI without flow control came along, but as I related above, it does not HAVE to be this way.  Roland and Yamaha designed hardware and wrote drivers that provided every bit as reliable USB-MIDI systems as the old MIDI-DIN system.  RS-232C was originally just the dial-up (land-line phone) modem protocol, before the MIDI guys got ahold of it and bent it to fit their needs, but it already had the flow control in it, so they did not have to do anything special for that.  It had the added advantage of opto-isolators at each end of the comm line, so no groumd hum that USB-MIDI can suffer from, especially with these transformerless switching power supplies that tie one side of the commercial AC power line directly to your delicate audio equipment.

 

You are correct with the work-around for "generic" USB-MIDI.  Keep the SYSEX messages in small chunks.  I also doubt that Marek would have a problem with SYSEX messages for most of the Casio models, unless he started stringing them together in large bundles, which a lot of programmers tend to do.  At least he is now aware of the possible problem, and will not have to re-invent the wheel, trying to figure out what to do about it, if he does run into it.

 

- T -

 

[markonis]

@shiihs As I know of the issue with parameter values larger than 127 I'm using midi toolkit which let's me handle conversion from 7bit to 8bit bytes and back.

 

@- T - Since I'm currently in the initial phase of analyses / development I just did only few tests of bulk data transfer - it went ok but the data volume was small. However, I'm quite sure that I'll be able to handle the potential issue with control flow of sysex messages. Let's see:

1) I'm already implementing handshake bulk data transfer method which according to CASIO specification is the most reliable method. It enables me to separate one large data transfer session into multiple subsessions where within each subsession only one packet of data is transferred. Each I/O device involved in the process has to acknowledge both correct sending and receiving of data. In case of an error the keyboard re-sends the packet  (the number of re-tries is programmable ). In addition to this I have the possibility to postpone the transfer process until the bottlenecks get cleared.

2) the size of a transferred packet ( minimal and maximal number of bytes included in a packet ) is stored within keyboards internal parameters which can be programmatically redefined ( when I was reading about this option in a specification I didn't know why I would do it but thanks to your comments I do now ),

 

3) the midi toolkit that I'm using  has a FIFO buffer functionality already implemented so I can use it and set the buffer's low / high filters for the cases of buffer over / under flowing. 
But as @- T - has said I also think that I'll not encounter this issue. On the other hand, it's always good to know beforehand about the potential problems and have proper tools available to handle them.
Thank you, guys, for your inputs since now I can think through the transfer process in detail and choose the most reliable solution.

 

MarekP

[Jokeyman123]

To shiihs and markonis...how are you progressing with these projects? Any new developments? Just curious-Python is a pretty unusual programming language from my limited windows Visual Basic and C++ knowledge. I'm afraid i couldn't offer much help.

[markonis]

Hi Jokeyman123,

 

I'm glad you're asking. For some time I was about to post an update about the state of my app but I was somewhat hesitating since I was stuck and depressed about the progress. This has changed today at 7 PM when a major breakthrough happened and after few months I finally managed to solve one important piece of a puzzle. Let me explain.

 

The Tone Editor module ( first and pivotal one ) that I'm working on covers two major functionalities.
1) let user modify all tone's parameters in real-time using either UI or entirely through the instrument using the sliders and/or dial knob

2) let user read all existing tones ( preset and user ) parameters and save modified tone as new / existing one.

 

Point one is finished and real-time modifying tone is working correctly for basic tone's parameters ( non-DSP ). In reality this means that I can choose any tone and using sliders I can modify it and hear instantly the effect of the change.
But there's a catch. To get the details of the selected tone I have to send proper sysex MIDI message to instrument which should respond with requested info ( i.e. a list of its parameters ). And here I've struggled for months. I was sending correct request, I was receiving correct response ( according to a specification ) which I verified through various SW tools but I was unable to see the same values of parameters as these which I've stored as my user tone.
I was only able to identify that some values have changed in comparison with a referential, clean tone which I've prepared beforehand.
I manipulated the received data in multiple ways ( for.ex 7bit to 8bit conversion ) but no success - even the number of received bytes didn't match the count specified in the header of received message. But today I've solved it.

 

What this mean is that now I can extract all parameters for all tones and after modifying these parameters I can save the tones as user tones ( since now I know the position of each param value in a keyboard's user area ).

One thing is still not clear to me, though. When I'm selecting specific DSP effect of a selected tone through a keyboard CASIO doesn't send any midi message, however when I modify any of 7 parameters related to this selected DSP effect I'm getting the midi messages. It means that I can see that I'm changing for. ex. 3rd parameter of currently  selected DSP effect but since I don't know what DSP is selected I can't figure out what that 3rd parameter is ( for. ex Chorus Depth ).

The other side effect of this is that when I'm modifying tone's basic parameter using the app and sliders I'm hearing the effect instantly. However, when I'm modifying the DSP parameter of a tone I don't hear the change since I don't know what midi message to send to a keyboard ( it's not a standard midi channel message but most probably specific sysex IPR/IPS message = "Individual Parameter Receive/Send" message that is related to a modification of a DSP's parameter ).

I'm playing with it now but when I'm requesting info about such a DSP's parameter of currently selected tone I'm receiving information about some specific preset DSP's parameter instead.

 

In other words, when I'm extracting all tones' parameters ( including DSP's ) I'm able to correctly interpret these parameters and its values.
But when I'm modifying DSP params of a currently selected tone using sliders I don't hear how it sounds ( as opposed to when I'm doing the same through a keyboard ).

 

With all that said I think, that now I can see the light in the tunnel ( as we say in Slovakia 🙂 ) and I'm nearing the phase when I'll be able to release first version for testing.

This is all for now.

As always, any comments are welcome.

 

MarekP
 

 

[cluelesstoo]

I have sone considerable work determining file formats for WK-7500/WK-7600, primarily with tone (.TN7)

files and DSP (.DS7) files, using a reverse engineering approach, as follows:

1) Job 1 is to get all tone files stored on my computer's hard drive. This requires the tedious

mind-numbing process of loading each of the 800-odd tones into the User tone area and transferring them

from there to my hard drive using Data Manager.  Like:

a) Load the 35 tones in group A (piano) into User tones 1-35

    Save these 35 User Tones to my hard drive in directory A

Some groups (like H) have more than 100 tones, so I load the first 100, save them,

and then load the remainder.

I will attempt to attach a compressed (rar_TN7.rar) file containing all WK-7600 tone files.

The format of the tone files can then be discerned by observing the minimum, maximum, and variation

of each byte in these files, and by using the tone editor to make changes and observe how each change

affects the binary contents of the tone file.

Finally, I attach the source code of a visual basic program I use to manipulate tone and dsp files,

which contains the complete format of tone files and dsp files. It turns out that the latter portion of

each tone file is in the exact format of a dsp file.

The program contains a variety of utility routines, such as using a single tone file with pleasing effects,

and applying those plesing effects to each of the 100 User tones in a second or 2.  I maintain several

sets of 100 user tones (stored as PK7 files and can load any such desired set from my computter to the WK-7600-

within about 20 seconds. I will also attach a number of these PK7 files, each containing 100 user tones.

 

Form1.vb REK100.PK7 REK100CP.PK7 WK-7600_170911.PK7 WK-7600_170911CP.PK7 WK-7600_200301.PK7 WK-7600_200302(2).PK7 WK-7600_200302.PK7 WK-7600_200303.PK7 WK-7600_200303CP.PK7

_TN7.rar

[Jokeyman123]

Amazing work, please keep working on this. A real editor librarian for the Casios that don't have one would be pretty popular I think. i would beta test on whatever Casio I think would be able to respond to the software. The XW, PX5s and even my old PX575 have some pretty impressive software front-ends, but that's where it seems to end, to my limited knowledge at least. You both  (Markonis and cluelesstoo) must already be aware-any Casio tones as you might have seen from the binaries-have specific DSP effects as part of the tone itself-rather than the somewhat more traditional design of applying an external DSP to the tone, as a completely separate program. These Casios can do both, within limitations-call up their interior or embedded DSP with the tone, and apply another DSP to it in the signal path-which can create some confusion for the player.  This fact got by me when i first started trying to program and edit user tones. This may only be true of certain tones though, I have not studied or formulated which tones specifically do this. I am referring to the later Casios-such as the 350, 560 and probably the s1000/3000. i am not sure if the older AHL or even older ZPI follow this design for DSP's and tones, i have only been able to go by what I hear.

[Just Alex]

Not exactly this topic related, but does anyone tried to load available firmware into IDA32 pro or something similar, to determine type of CPU, offsets and so on?

[BidiMc]

@cluelesstoo

 

I'm not very familiar with visual basic. Can you tell me how best to run this form1.vb?

Should I use visual studio vb.net or another way? renaming to .vbs and running directly in Windows produces error on first line.

 

I'm considering creating some renamer as requested above based on your code. 

Looking for an excuse to learn Python  And go from there.

 

EDIT: ah, I see there is no Main. This is just a class. but ok, should be enough to get me started.

 

@Chandler Holloway what would this renamer tool look like? How would you like to use it? Can it be on the commandline something like rename_ac7 <file> <newname>?