Managing edits to sys files across many projects & platforms

[Ridgeglider]

rnixon's recent comment http://forum.embeddedethernet.com/viewt ... ?f=4&t=834 inquiring about whether hector had made changes to system files that prevent successful operation brings up an issue that I've been wondering about recently. Maybe some of you have some suggestions? I work on quite a few projects that might utilize three or four different NB platforms in a given month. Many of the projects require changes to the system files: TICKS_PER_SECOND, enabling long file names, Task Prios, max_number of tasks, buffer sizes, stack sizes, timeouts, fast-stack setups, ucos_stackchecking, multi-home, udp_fragments, etc.

What works and is appropriate for one project is usually not OK for another. Currently, to my knowledge, there is no easy way to document and record what works for given project, and no way to apply it without doing a system rebuild that will then affect all future other projects. There must be a more effective way to record these settings in an individual project and to allow other projectss to be unique and appropriately different.

How have others dealt with this? I may be wrong, but I don't think having unique, local versions of the two main configurations files (predef.h and constants.h) in each project works (while also retaining the originals in the nb/include directories). If so, perhaps I just need to do a rebuild all each time I work on that project for a while? Suggestions?

[rnixon]

Good points, and I'm not trying to hijack the thread, but I have always wondered about the posts here regarding changing ticks per second. As near as I can tell, all that will do is increase system latency time and using a separate timer is a an alternative to be considered with micro second or better resoltion. If I need a faster timer, I use one of the (usually many) timers available on the processor.

So I can understand some of the things like fast task stacks, and even priorities (if I need more than 20 tasks!) but that usually doesn't some up for me. So I guess my question is one of architecture, can I avoid a change to the system libs is I use a select() in one task, rather than many tasks, use a h/w timer rather than change ticks per second, etc. If that reduces the number of time I need special system file changes, then maybe I can have just two or three, renaming directories?

I do think the system task priorities should have been shifted much further down the range, since any task I create almost always need network services.

[Ridgeglider]

I also question the need to alter TICKS_PER_SEC, and agree that the h/w timers are great for many or most timing tasks. I agree that I shift most of the system tasks down by 30 to allow more user tasks access to working system resources:

Code: Select all

#define TASK_OFFSET        ( 30 )
#define HTTP_PRIO          ( 45 - TASK_OFFSET )
#define PPP_PRIO           ( 44 - TASK_OFFSET )
#define WIFI_TASK_PRIO     ( 41 - TASK_OFFSET )
#define TCP_PRIO           ( 40 - TASK_OFFSET )
#define IP_PRIO            ( 39 - TASK_OFFSET )
#define ETHER_SEND_PRIO    ( 38 - TASK_OFFSET )
#define WIFI_TASK_PRIO     ( 37 - TASK_OFFSET ) 

That said, there are still plenty of changes unique to certain projects, again: Max number of tasks, buffer sizes, fast_stack, udp_fragments, long filenames, to name the most commonly altered in constants.h, along with the ucos_stackcheck and tasklist options in predef.h. I wonder how to do this in a more project-specific way w/o the need to rebuild system files which in turn always makes me wonder about the unanticipated effects on how doing so might affect other projects. Currently, I have a text file in many projects that is a cheat sheet for how I need to (manually) set up the system files before a rebuild and subsequent compile. This does not make it easy to jump between projects, so I keep thinking there must be a better way.

[rnixon]

I like your task offset idea. That might even be a good thing to have in the constants.h file in the release, just so its more obvious to those not used to modifications like that. I don't have a good answer to how to maintain multiple system files modifications.

[tod]

I just use the standard system build from NB and haven't had to tweak it on a project by project basis. I mention that first so you know I have no practical experience in this area. If I were to customize the system on a project basis I would want to set it up the same way I do for my non-embedded projects. I would want to have all the files I need in my source code management system. When you make your customizations and rebuild the system, does it just update all the library files in the C:\nburn\lib folder? If so I would copy those over to my project folder so that I have both a src and lib folder for the project. Then I would adjust the linker settings to have a path to that folder and remove the path to nburn\lib. Of course since I've never done this I'm not sure this works but it's where I would start. I would do the same thing with any modified include files. It would probably be easiest to move the entire nburn/include folder over into an include folder for the project and update the Paths and Symbols section of the project to use this folder. You'll still be dependent on the rest of the NNDK that was used for the project. If your SCM hard disk space is cheap you could always include a copy of the NNDK with the project (again it would only get put in there once, so it's a one time hard disk hit). Otherwise, a readme.txt that indicates which NNDK was used and where to find it would suffice.

Just thinking out loud...

[Ridgeglider]

Thanks for your response Tod. I am wondering how Forrest and Larry might recommend managing these details? For a long time I resisted modifying the system files exactly because of the issues described in this thread. However, if you look at the number of times folks (particularly the NB guys) suggest one of these system edits in this forum, it's clear that although there are solid advantages to the edits, it can become tricky to know how a particular project build must be configured, particularly if you are coming back to a it a while later. This gets worse after you've performed sytem updates. Since this uncertainty makes me very uneasy, I'm looking for a more systematic and transparent approach, but ideally one that does not require too many subtle edits to the path and symbol settings for each project. Don't get me wrong, I like the idea of optimizing the system for a given project since each has unique constraints. However it would be great of there was documentation or a method to keep track of it more clearly.

[lgitlitz]

I agree that it would be much better to have all the system configuration files in each project. The way the current system is set up I do not see an easy way to change how this works. I will definitely bring this up at the next meeting. One problem is that many of these things such as OSStackCheck will require rebuilding of the libraries. There is really no way around this since more code is inserted to keep track of all the task stacks. If we made this a run-time option with if statements then all release builds would grow in size and have a minor hit in performance.

I know this not the most elegant solution but why not just use a batch file to automate the saving and restoring of the system files and configurations? I put together two batch files you can try out. To use these just put them in an empty folder anywhere on your pc and set the NetBurnerPlatform definition at the top to each file to the platform you are using.

The "SaveSystemConfiguration.bat" will save all files and folders from the ..nburn\system, ..nburn\include, ..nburn\%NetBurnerPlatform%\system, ..nburn\%NetBurnerPlatform%\include, the ..nburn\%NetBurnerPlatform%\PROPERTIES file and also the ..nburn\lib\FatFile.a. Folders will be created in the local directory where the batch file is located to store all these files, no object files are saved.
The "RestoreSystemConfiguration.bat" will first copy all of the above directories into ..nburn/BACKUP/. It will then delete the existing versions from the nburn directory and copy in the saved versions from your local directory. After that it will go through and clean and rebuild the release and debug libraries for both ..nburn\system and ..nburn\%NetBurnerPlatform%\system. If the builds fail then the batch will stop and display where it failed, prompt you to restart the build.

If you do not want to rebuild your files when you switch between projects you could change the batch files to simply save and restore the NetBurner.a, DBNetBurner.a, %NetBurnerPlatform%.a and DB%NetBurnerPlatform%.a libraries from the lib directory. This should be much simpler then the current restore batch I provided. Personally I like to see everything rebuild properly when changing the system files. I recommend that you run this on an "originals folder" after a fresh install so you can always revert back to the default system configuration. You will not be able to use this to set the configurations of a new version of the NNDK from an old version since it will overwrite everything.

-Larry

[lgitlitz]

Forgot to mention these were tested on a Win7 machine. I am not sure if all the commands are compatible with all older versions of windows. These batch files are safe but NEVER run a batch file from the internet without viewing it in a txt editor first. A single "evil" line in a batch could do some major damage to a PC by deleting/moving files or even sending data over the network.

-Larry

[roland.ames]

Here is a suggestion:

The project properties include a checkbox "use project-specific system files".
When it is set, the relevant system source / lib files are copied into a subdirectory of the project, the include and library paths are adjusted accordingly. When "rebuild system" is activated, the local project specific system files are rebuilt and/or the project make would include rebuilding system files if necessary.

This gives each project the option of using the common system files under the NBURN directory, or of having its own local copy of the system files as part of the project. The project-specific system files can then be archived and/or checked into revision control as part of the project source.

[Ridgeglider]

Larry, thanks for your batch file ideas. I will give them a try. They might allow recreating a particular build, but they are not exactly transparent about the differences that make a particular build unique. For example, there would be no easy way to tell which system files were modified without diffing the entire system through all its individual directories. The recent post by roland.ames sounds like the beginnings of a great idea w/ the least hassle. However, an additional, and I believe critical component would be to include a way to know which files have been modified from the originals so that they can receive special attention during any update of the NB platform.

Perhaps a solution to both might be that if a "use project-specific system files" box were checked in the project properties, a subsequent tab might include a list where you could fill in the names and original path to the uniquely modified files. It seems to me, (without intimate knowledg of the make utility) that only those source files then need to be copied into the local project. Perhaps each project would have a directory called "LocalSysFiles". Of course it would also need directories for any compiled or linked output resulting from them the modified system files. The build paths would clearly need to be adjusted to reflect the local source and resulting object files/libraries.

After a system update, any projects incorportating local (modified) system source files would throw a warning asking the user to manually diff the (old) system files in the local project against their counterparts in the new "originals" directories. Maybe, to aid in flagging these instances, the system files need a first line token indicating which NB system revision they are part of to help in catching the old files with customized changes. If there is an old revision, compilation throws a warning or error?

Usually, in the case of constants.h and predef.h, (the most likely candidates for these edits and mods) the diff/edit process would be pretty quick once the old files werre identified. I think this concept builds on what Tod was suggesting a few posts ago, although this strategy might elevate it to a sort of standard procedure instead of being something only a few cognoscenti were capable of. I think this kind of facility would be a really useful improvement.