0

I am using a testing framework to test designs written in VHDL. In order for this to work, a Python script creates several "libraries" and then adds files in these libraries. Finally the simulator program is invoked, it starts up, compiles all the files into the specified libraries and then runs the tests.

I want to make changes in the way we specify what "libraries" to create and where to add the files for each library from. I think that it should be possible to write the description for these things in JSON and then let Python script process it. In this way, the same Python script can be used for all projects and I don't have to worry about someone not knowing Python.

The main issue is deciding how to express the information in JSON file. The JSON file shall create entries for library name and then location of source files. The fundamental problem is how to express these things using some type of pattern like glob or regular expression:

  1. Pattern for name of folder to search
  2. Pattern for name of subfolders to search
  3. Express if all subfolders should be searched in a folder or not
  4. What subfolders to exclude from search

This would express something like e.g "files in folder A but not its subfolders, folder B and its subfolders but not subfolder X in folder B"

Then we come to the pattern for the actual file names. The pattern of file names shall follow the pattern for the folder. If same file pattern applies to multiple folders, then after multiple lines of folder name patterns, the filename pattern applying to all of them shall occur once.

  1. Pattern for name of file to add into library.
  2. Pattern for name of file to exclude from library.

This would express something like e.g "all files ending with ".vhd" but no files that have "_bb_inst.vhd" in their name and do not add p.vhd and q.vhd"

Finally the Python script parsing the files should be able to detect conflicts in the rules e.g a folder is specified for search and exclusion at same time, the same files are being added into multiple libraries e.t.c. This will of course be done within the Python script.

Now my question is, does a well defined pre-existing method to define something like what I have described here already exist? The only reason to choose JSON to express this is that Python has packages to traverse JSON files.

quantum231
  • 2,420
  • 3
  • 31
  • 53

1 Answers1

0

Have you looked at the glob library?

For your more tricky use cases you could specify in/out lists using glob patterns.

For example

import glob

inlist_pattern = "/some/path/on_yoursystem/*.vhd"
outlist_pattern = "/some/path/on_yoursystem/*_bb_inst.vhd"

filtered_files = set(glob.glob(inlist_pattern )) - set(glob.glob(outlist_pattern))

And other set operations allow you to perform more interesting in/out operations.

To do recursive scans, try ammending your patterns accordingly:

inlist_pattern = "/some/path/on_yoursystem/**/*.vhd"
outlist_pattern = "/some/path/on_yoursystem/**/*_bb_inst.vhd"

list_of_all_vhds_in_sub_dirs = glob.glob(inlist_pattern, recursive=True)

With the recursive=True keyword option, the scan will be performed at the point in the path specified, and where the ** notation is used, plus zero or more subfolders, returning the files that match the overall pattern.

Thomas Kimber
  • 10,601
  • 3
  • 25
  • 42
  • I have not looked at it in detail yet, I am new to Python myself. – quantum231 Apr 08 '21 at 07:53
  • Sure, give it a try - it allows you to set the patterns using a [fairly common asterisk/wildcard formatting template notation](https://en.wikipedia.org/wiki/Glob_(programming)). For the ins/outs and other logical layers you want to apply, you might be able to use python sets to apply specific, named filters. – Thomas Kimber Apr 08 '21 at 07:57
  • so you mean use glob pattern for each pattern of file or folder to be included or excluded? Should I prefer glob or regular expression? – quantum231 Apr 08 '21 at 08:49
  • Yes, build lists (sets) of filenames that match wanted/unwanted patterns and use set operations to end up with what you want. Re glob vs regex, that's up to you - the advantage of glob over regex is that glob will do the file traversal for you without having to do `os.scandir` type loops - if you find yourself doing more of that, then the advantages of glob over regex are reduced. You can use glob recursively though, so that's an option - see: https://stackoverflow.com/questions/2186525/how-to-use-glob-to-find-files-recursively – Thomas Kimber Apr 08 '21 at 09:17
  • alright, so in your example I can see you using pattern for file, what about pattern for folders, or subfolders for recursive search? – quantum231 Apr 08 '21 at 09:22
  • Folders and files are returned if their text-name matches the input pattern - so the default case finds files or folders without making any distinction. For the recursive search, use the keyword parameter `recursive=True` or try using the `**` syntax. It's all there in the linked documentation. Each pattern will return a list of matching filenames or folders that you will then need to decide what to do with. – Thomas Kimber Apr 08 '21 at 09:26