markonis

Hi Jokeyman123, I'm glad you're asking. For some time I was about to post an update about the state of my app but I was somewhat hesitating since I was stuck and depressed about the progress. This has changed today at 7 PM when a major breakthrough happened and after few months I finally managed to solve one important piece of a puzzle. Let me explain. The Tone Editor module ( first and pivotal one ) that I'm working on covers two major functionalities. 1) let user modify all tone's parameters in real-time using either UI or entirely through the instrument using the sliders and/or dial knob 2) let user read all existing tones ( preset and user ) parameters and save modified tone as new / existing one. Point one is finished and real-time modifying tone is working correctly for basic tone's parameters ( non-DSP ). In reality this means that I can choose any tone and using sliders I can modify it and hear instantly the effect of the change. But there's a catch. To get the details of the selected tone I have to send proper sysex MIDI message to instrument which should respond with requested info ( i.e. a list of its parameters ). And here I've struggled for months. I was sending correct request, I was receiving correct response ( according to a specification ) which I verified through various SW tools but I was unable to see the same values of parameters as these which I've stored as my user tone. I was only able to identify that some values have changed in comparison with a referential, clean tone which I've prepared beforehand. I manipulated the received data in multiple ways ( for.ex 7bit to 8bit conversion ) but no success - even the number of received bytes didn't match the count specified in the header of received message. But today I've solved it. What this mean is that now I can extract all parameters for all tones and after modifying these parameters I can save the tones as user tones ( since now I know the position of each param value in a keyboard's user area ). One thing is still not clear to me, though. When I'm selecting specific DSP effect of a selected tone through a keyboard CASIO doesn't send any midi message, however when I modify any of 7 parameters related to this selected DSP effect I'm getting the midi messages. It means that I can see that I'm changing for. ex. 3rd parameter of currently selected DSP effect but since I don't know what DSP is selected I can't figure out what that 3rd parameter is ( for. ex Chorus Depth ). The other side effect of this is that when I'm modifying tone's basic parameter using the app and sliders I'm hearing the effect instantly. However, when I'm modifying the DSP parameter of a tone I don't hear the change since I don't know what midi message to send to a keyboard ( it's not a standard midi channel message but most probably specific sysex IPR/IPS message = "Individual Parameter Receive/Send" message that is related to a modification of a DSP's parameter ). I'm playing with it now but when I'm requesting info about such a DSP's parameter of currently selected tone I'm receiving information about some specific preset DSP's parameter instead. In other words, when I'm extracting all tones' parameters ( including DSP's ) I'm able to correctly interpret these parameters and its values. But when I'm modifying DSP params of a currently selected tone using sliders I don't hear how it sounds ( as opposed to when I'm doing the same through a keyboard ). With all that said I think, that now I can see the light in the tunnel ( as we say in Slovakia 🙂 ) and I'm nearing the phase when I'll be able to release first version for testing. This is all for now. As always, any comments are welcome. MarekP

@shiihs I hesitated if I should reply and post a comment on your topic but then the discussion really went off this topic. Sorry for that, I didn't mean to advertise my tool here. Few month ago I've started a topic for my tool ( WK 7600 - Structure of file formats ( i.e. AC7, TN7, RM7, AL7 ) in the form of a question / request for a help. Is it possible to move the above conversation to my topic? Regarding the portability to Linux: I'm a whole-life Windows developer so Linux is unknown to me. From what I've read about Mono and NET Core I would probably vote for Mono with a thorough testing. However, since I'm dealing with a low-level functionality strongly dependent on a HW device plus specific user GUI biased to this HW functionality I see it very difficult for me at this time. I have some ideas on how to do it but I agree that we should discuss it within my topic.

Correction: it seems to me now that a list of models supported by the current version of a tool is broader - CTK6200 / CTK 7300 / CTK 7200 / CTK 7300 / WK 6600 / WK 7600.

I'm afraid that at the start of the beta-testing we will need to concentrate on WK 7500/7600 and CTK 7000/7200 models since MIDI implementation of sysex messages seems to be the same. The main differences between the above models ( from the tool's perspective ) are "only" the number of keys and a user storage available per data structure ( user tones, user DSP effects, reg. bank slots, ... ). As I already quickly studied MIDI implementation of some other CASIO models I can see that the structure of sysex messages differs per model. I'm building the framework so that future expansion / incorporation of other models is possible but first I need to test the functionality on a keyboard that I have at home ( WK 7600 ). That doesn't mean that we can't test the tool on other models but I'm 90% sure that it'll not work. The worst case will be if internal structures of CASIO objects will differ - in that case I will need to adjust most of the framework, i.e. make it generic. Which is doable, but definitely another project.

I like to develop SW in Visual Studio - my favorite languages for Windows / web applications are C# / ASPX.NET. Within a data warehouse projects ( which is my core business ) I work with MS SQL Server and Oracle databases ( i.e., SQL and T-SQL languages ).

@Jokeyman123: As you've correctly pointed out the deep built-in tone/dsp editing functions available on WK / CTK models are not easily editable through the keyboard. There are multiple pages per tone and only few parameters on a page can be edited at one time. That was the main reason for me to start creating the software that would make this process easier. When I had the basic GUI for tone editing feature ready it kind of reminded me a standard VST plug-ins which are freely available on the web. The difference is that VST plug-ins try to be universal and thus they offer "only / mostly" standard MIDI functionality ( i.e. standard sysex messages ). What I'm trying to build is a custom made tool specifically tailored for CASIO WK/CTK models ( i.e. using also provider-specific sysex messages which would enable us to use maximum of features / functionality of a keyboard ). As @shiihs correctly points out the MIDI features implemented in CASIO keyboard are documented quite well in the specification but it's a lot of work to extract and convert inner data structures used by the keyboard to GUI controls. There are few obstacles in this process that I'm currently dealing with: 1) the CASIO allows to send only user tones - not built-in tones ( note: each user tone is based on one built-in tone ). 2) the CASIO sends parameters for a tone as a block of bytes. There are few parameters that are common for all tones, but many tones ( mostly dependent on DSP settings ) have variable structures ( for ex. different number of parameters per tone, different number of bytes per parameter, ... ). 3) as a result of an obstacle 2) the number of bytes received per tone ( = number and values of tone parameters ) differ for each tone 4) CASIO doesn't send a specific sysex just for a movement of a slider / knob, it sends it only as a reaction to change of some tone's parameter => I need to ensure that the keyboard is set in a special way - as if a user changes some parameter ( let's say tone velocity ) with the slider. => but, at the same time I need to ensure that a correct tone parameter is changed in the tool as a response to a slider / knob move and that GUI reflects the change properly. So, to overcome the obstacle 1) and get the proper list of parameters for a built-in tone I need to manually change some parameter of a built-in tone on the keyboard and save this changed tone as a user tone - this way I can export all the parameters for any tone ( built-in or user tone ). The more difficult issue are the obstacles 2) and 3) - in order to find out a specific parameter of a tone ( + where it's located in the array of exported bytes ) I need to change the relevant parameter on the keyboard, save the tone, export it and compare all the changed bytes with the original tone parameters list ( bytes ). In the attached screenshot you can see that number of Tone Basic parameters 1 and 2 ( + Tone DSP parameters 1 ) is fixed but the number of Tone DSP parameters 2 is different for each tone and at this time I don't have the map available for all the tones and it's ( variable ) parameters. Of coarse, these variable tone parameters and its values need to be generated dynamically in the tool ( when the tone is selected ). If we take into account that there are 700 built-in tones it's obvious that the number of combinations of tones and its parameters is quite huge 🙂 ( as a side task, I'm working on a way how to automate an extraction and a proper mapping of these parameters ). The good think is that I have ( roughly ) finished the framework so that I can now concentrate on the first module's ( = Tone Editor ) main functionality. Your comments are welcome.

Hi Rossier, The keyboard alone doesn't have such functionality ( I didn't found it either ), that's why I'm currently building an application that should provide me with an ability to achieve this. However, it'll take few month until I'll have first version of the app ready for testing. Pls, see the discussion here. https://www.casiomusicforums.com/index.php?/topic/16912-wk-7600-structure-of-file-formats-ie-ac7-tn7-rm7-al7/ Best Regards, MarekP

@shiihs As I know of the issue with parameter values larger than 127 I'm using midi toolkit which let's me handle conversion from 7bit to 8bit bytes and back. @- T - Since I'm currently in the initial phase of analyses / development I just did only few tests of bulk data transfer - it went ok but the data volume was small. However, I'm quite sure that I'll be able to handle the potential issue with control flow of sysex messages. Let's see: 1) I'm already implementing handshake bulk data transfer method which according to CASIO specification is the most reliable method. It enables me to separate one large data transfer session into multiple subsessions where within each subsession only one packet of data is transferred. Each I/O device involved in the process has to acknowledge both correct sending and receiving of data. In case of an error the keyboard re-sends the packet (the number of re-tries is programmable ). In addition to this I have the possibility to postpone the transfer process until the bottlenecks get cleared. 2) the size of a transferred packet ( minimal and maximal number of bytes included in a packet ) is stored within keyboards internal parameters which can be programmatically redefined ( when I was reading about this option in a specification I didn't know why I would do it but thanks to your comments I do now ), 3) the midi toolkit that I'm using has a FIFO buffer functionality already implemented so I can use it and set the buffer's low / high filters for the cases of buffer over / under flowing. But as @- T - has said I also think that I'll not encounter this issue. On the other hand, it's always good to know beforehand about the potential problems and have proper tools available to handle them. Thank you, guys, for your inputs since now I can think through the transfer process in detail and choose the most reliable solution. MarekP

Thank you very much for this valuable information. I'll take it into account and test it properly.

@Chandler Holloway All the above applies to models WK xxxx / CTK xxxx since structure of sysex messages and internal allocation in keyboards data in memory for other models differ. In order to make the app usable by other models I would need to have the models at my disposal to test them ( the good thing is that technical specification of MIDI implementation for the other models is available online ). From what I saw in specs for different models I personally think that the internal data could be very similar for different models but that would need to be verified and tested. At this stage I need to build a proof of concept in the form of an app for WK 7600 which I have at home. After that I can think how to make the framework universal and workable also for other models. MarekP

@Chandler Holloway Thanks for the perfect guide regarding the structure of CKF files and applying markers to MIDI files. I was searching for this information for quite a long time. @shiihs I was looking at the "edisyn" framework but it's not what I need for these reasons ( the following is not a critique just a summarization of facts ) : 1) it seems to me that the framework tries to be universal tool so it only utilizes program changes (PC=tones) and controller changes (CC=effects) for GM 1 specification so that broad range of keyboards from different manufacturers can use it ( though the framework recognizes only CASIO CZ series - as per user guide ), 2) maybe I could incorporate other CASIO models' ( via specific sysex messages and PCs / CCs ) using the framework but I'd need to learn the specific language/syntax to do the same what I'm doing in my project right now, 3) the most important: the functionality of the framework looks like just another VST-like tool/app for editing tones but I'm planning to build something more specific, broader in scope and targeted primarily for CASIO keyboards ( see more details below ) @shiihs I'll probably disappoint you since I'm a whole-life Windows developer so I'm not able ( at least now ) to build a cross-platform framework. However, I'm planning to build 2 versions of the app: desktop and mobile ( for tablets ) so with a little investment also Mac users could benefit from my app ( is this feasible for you as an end user? ). @shiihs Your idea to create AC7 parser looks interesting to me. Based on my current analysis I have the following possibilities regarding access to internal CASIO data ( there are more but for now I'm focusing just on these ones): 1) import / export of user tones - YES, as per single tone ( built-in tones - NO directly, YES indirectly - see below ) 2) import / export of user DSP effects - YES as per single DSP effect ( built-in DSP - NO directly YES indirectly - see below ) 3) import / export of user registration banks - YES, all in one block ( need to verify ) ( built-in RB - NO directly YES indirectly - see below ) 4) import / export of user rhythms - YES as per single rhythm ( built-in rhythms - NO directly YES indirectly - see below ) 5) import / export parameters of points 1) - 4) + mixer parameters 6) import / export of all data ( = AC7 file ) - export YES, all in one block, import YES only in case of being able to parse and identify each byte of data in the block All of the above enables me to build an application with modules ( i. e. editors ) for each of the above points. In addition to user data, I'm able also to export built-in keyboards data but it's quite a tedious process - I need to manually save each built-in user data ( for ex. tone ) as user data ( for ex. user tone ) and then export it. All the parameters for user tones and DSP effects can be exported per item ( for ex. tone ) but reg. banks as well as all user data ( = AL7 file ) can be exported only as one continuous block of data which makes parsing it a lot more complicated. Since I'm currently not able to parse the user rhythm data ( I don't know the internal structure of the data, for ex. where the rhythm sections start ) the user rhythm parser of AC7 file ( or at least CKF file ) would be of great help for me. Regarding user rhythms, at this stage of analysis I'm planning to build an easy MIDI file parser which should enable user to set markers in the imported midi file and copy & paste parts of midi file into separate rhythm sections + modify channels + apply effects to the tones so that it complies with CKF / AC7 file ( of course I'm not going to create another Anvil Studio - for complicated adjustments there are many good applications ). Such a modified midi file should then be processed via IDES application into proper CKF / AC7 file and imported into keyboard as a user rhythm. So it seems to me that both me and you, shiihs, are trying to handle the user rhythms but each of us from a different side. @Chandler Holloway Based on the above points I should be able to at least change names of reg. banks and user rhythms so that I can create kind of a map between these two user data. I think that this is a functionality that you were asking for in your contribution, am I right? As a working title I named this module in my app as a Registration Bank Editor. MarekP

Hi Jokeyman, The SysEx route in addition to all the midi events processing staff is quite challenging ( there's a lot to study ) but after few successful tests I'm now pretty sure I can do it. I was able to bulk export parameters for one specific CASIO preset tone from the keyboard using sysex's HBR ( Handshake Bulk Parameter Set Request ) transfer method. So the bulk data transfer is functional. However, I also need to send / receive individual parameters in order for the app to work correctly. But currently I can't manage to export any specific parameter using IPR ( Individual Parameter Receive ) transfer method - I'm still getting REJECTED error message from a keyboard. I will probably need to see some examples of such SysEx message - maybe some users in this forum were successful with this. I feel that I'm not specifying parameters "blk" / "idx" / "len" correctly when I'm building this type of a sysex message. The other issue which I need to solve is the markers for all the sections ( Intro, Variation 1, 2, ... ) in a CASIO rhythm. I will also need an example of a midi file with these markers specified. Until now I didn't find a list of CASIO rhythm section markers ( only for YAMAHA styles ). From what I have learned I know that the app will be able to cover these models ( CTK-6200 / CTK-6300 / CTK-7200 / CTK-7300 / WK-6600 / WK-7600 ). The differences between these models are basically: 1) number of keys ( i.e. octaves ), and 2) number of presets ( built-in tones / rhythms , user tones / rhythms, ... ). One issue I know of could be with keyboard models designed and shipped to Indian market. The mapping and internal numbers of built-in tones and rhythms will probably differ from those shipped to US/EU markets. In reality it could mean that when "Indian" user will choose let's say Grand Piano the keyboard will play some other instrument. The above paragraphs show that I will surely need some cooperation from users ( in the future ). That's why I was also thinking about getting few of the users involved in the development process as beta testers. I already see that the project is quite large and that each module will need to be developed separately with exception of the underlying framework that needs to interlink all these modules internally. As you can see the development will consume a lot of my private time ( I'm a full-time SW developer ) so I'll probably need to set some price for the app as a compensation. The beta testers will surely have some advantages as the testing will also take some time on their side. One more thing - as I'm a technical person ( not an artist ) the UI ( user interface ) of the app doesn't necessarily will have to be perfect. This is where a user-designed layouts can be helpful for me. My idea is to keep the look of the app as close to a keyboard one so that users can easily refer to documentation and/or keyboard LCD screen. So, when I'll have something working to show I will make the application available - we can then discuss details of a cooperation. Since the project is opened one and I'm willing to incorporate any good ideas for a functionality of the app any suggestions and recommendations are welcome. MarekP

Hello shiihs, Sorry for the long delay but I was unable to log in to my account. The website that Jokeyman123 pointed me to has brought me to a realization that I don't need to parse the files exported by CASIO keyboard. I found out that there's an official documentation for a WK 7600 describing all MIDI events that the keyboard is capable of sending/receiving. Via SysEx messages I'll be able to get to the most of the data needed for my project. So the answer to your question is that now I'm able to build the application though I'll need to go really low level that means quite a bit of tedious work in order get all the details. Now I'm in the phase of building the framework ( basic engine ) upon which I'll be able to build all the modules. MarekP

Hi Jokeyman, it is exactly as you've said - CASIO responded that the file formats is internal information not available to the public. I'm planning to develop a front-end for WK 7600 which would allow me to build the following functionality: 1) management of registration banks: 1.1 see reg. bank information - which rhythm, tones, settings is used for the particular reg. bank / slot 1.2 edit the slot info - play with different rhythms, tones, settings 1.3 merge multiple reg. banks - so that the user can save slots from different reg. banks alongside with a current loaded reg. bank 1.3 save the reg. banks info back to a file 1.4 re-upload the reg. banks to a keyboard 2) management of user rhythms: 2.1 create rhythm parts from a specific midi file ( for ex. copy a part of midi file and set it as INTRO, FILL-IN, .. ), 2.2 edit rhythm in real-time, i.e. re-assign tones / channels, mute channels, set volume and other available settings 2.3 save the user rhythms info back to a file 2.4 re-upload the user rhythms to a keyboard 3. management of user tones in real-time through front-end: 3.1 better use of sliders, mod wheel, ... for a manipulation of a sound/tone 3.2 modification of tone through keyboard is not so intuitive - using a front-end would be a quicker 3.3 save the user tones info back to a file 3.4 re-upload the user tones to a keyboard 4. manage sets of reg. banks, user rhythms, user tones 4.1 inter-connect the banks with rhythms and tones so that a user can have categories /sets ready for upload at once ( let's say dance category would upload specific reg. banks, related user rhythms, user tones suited for a dance songs, 4.2 re-upload specific user set to a keyboard in one step The most of the above functionality should work both ways - change in application should be immediately noticeable in keyboard and change in a keyboard setting should be reflected in the app. In addition to the above my idea is to build a website for this community of users that would allow them to exchange reg. banks, rhythms, tones, user sets since this is missing most in my opinion. All of this, however, strongly depends on the knowledge of the relevant file formats. MarekP

Hello guys, I'm a software developer and I'd like to build a few applications that will extend the functionality of this fantastic keyboard even more. For this I need to know the structure of the files mentioned in the title - that is, how the data in these files are physically stored and structured. I already contacted the CASIO company and I'm waiting for their response, however I'm trying to seek the answer also among you in case CASIO will not want to share it with me. Could you, please, point me to a right person, website, ... where can I get this info. Thanks very much. Marek