Kathleen Beaumont

Kathleen Beaumont

Forum Replies Created

Viewing 15 posts - 1 through 15 (of 28 total)
  • Author
    Posts
  • in reply to: Edit Engine and XML #12153

    Hi Stefan,

    I discussed this topic in the XMLExchange Plus help file. (See topic Understanding XML Dictionaries -> Dictionaries Are NOT Needed to Run EDITS -> The Edit Engine Still Expects a Flat Buffer.)

    Apart from the horror of all that would have to be redesigned in the Engine to accommodate this request, the multi-dimensional nature of XML would impact the logic of individual edits. For example, since Tumor data items exist in a many-to-one relationship to a Patient, the logic would have to be rewritten to loop through instances [1] to [n] of every Tumor item. This would ruin one of the major features of the EDITS software: its approachability for people who do not have programming backgrounds. Also, there are many thousands of legacy edits out in the field, still in use after decades, that somebody would have to rewrite.

    (I’ll attempt to attach a screen shot from the help file to this message; not sure if it will work.)

    The Engine should not be concerned with the data source. You don’t expect the Engine to know how to read and retrieve data directly from your database. The Engine’s job is to give you the tools (APIs) you need to know where it can expect to find each data item in the record buffer; it’s up to the calling software to prepare the data for processing accordingly.

    Of course, I’m a member of the old guard, so there is no telling what the future may hold!

    Kathleen

    Attachments:
    You must be logged in to view attached files.
    in reply to: Best practice for running Edits on XML files #11884

    Hi Stefan,

    Use whatever XML parser you like best. I needed something like XMLPlus.dll for my own work, and NPCR agreed to share it with anybody who wants to use it.

    If you decide to use an XML parser you already like in your Java work, do be sure that you understand all the requirements in the official NAACCR XML Implementation Guide. (That advice applies regardless of the parser you choose.)

    Converting the XML data file to a flat-format file is not necessary to run edits. The EDITS50.dll API includes functions and code samples showing how to build the flat-format buffer on the fly so that another disk file doesn’t have to be created (and eventually deleted). On the other hand, if you have other uses for the flat-format data, then converting to flat once makes sense.

    Building the data buffer dynamically also gives you more flexibility when data needs to be edited in more than one record layout format. (If I recall correctly, New York uses one or more custom layouts.) The XMLExchange Plus help file has a discussion about the advantages of creating an edit buffer dynamically, plus sample code (annotated C++) showing how to run edits on XML files.

    Please note that XMLPlus.dll was published with the NAACCR XML Implementation Guide was at v1.2. However, I have reviewed the change notices in the current version, and to the best of my knowledge, XMLPlus.dll conforms with the current Implementation Guide (v1.4).

    Kathleen (still retired, but clearly still interested)

    in reply to: Help with callback functions #10810

    EDITS50.dll is built with Visual Studio 2017, Platform Toolset Visual Studio 2017 (v141).

    I have been running searches on DuckDuckGo using key words C++ DLL Windows Server 2012 and variations, and finding discussions concerning Windows API function LoadLibrary and its (possible) dependence on the operating system’s C runtime library.

    So it would be very helpful to know where in the source code the application is failing, and what error messages are displayed (if any). This might involve embedding more std::cout statements in the source code.

    I do think the problem has to do with the environment, and perhaps adding one or more OS libraries to the system directory (e.g., C:\Windows\SysWOW64 or its Win2012 equivalent). If you have a network admin, could (s)he help you debug this?

    in reply to: Help with callback functions #10809

    Unfortunately, I cannot be of further assistance, for the following reasons:

    1. I do not have access to a Windows 2012 server.
    2. I am retired; I just subscribe to this forum in the spirit of “peer support”, in case I can easily answer a question posted here.

    You might try contacting NPCR directly, to see if they can help you. Email to cancerinformatics@cdc.gov

    Does the test application work at all? If so, how far does it get?

    in reply to: Help with callback functions #10807

    Sorry, I don’t have any suggestions regarding why your application would run in one environment but not another.

    As a test, could you install EditWriter-64 bit on your 2012 server and use it to run edits? (Use the Edit Set/run edits feature of that program to generate a report.) If that fails, it could indicate something is needed in your server environment. (EditWriter is written using C++11 and does not require any supporting frameworks.)

    Perhaps somebody else reading this forum who works with Win2012-64 will offer some insight.

    in reply to: Help with callback functions #10805

    The EDITS50 API documentation provides C++ code samples for all functions. Go to the page for Edit_RunEdits, and follow the link at the bottom for C++ code sample.

    Additionally, since you are writing a console app using Visual Studio, you could retrieve the archive containing the Linux binaries from the EDITS50 download page (https://www.cdc.gov/cancer/npcr/tools/edits/edits50.htm – scroll to the bottom of the page and download EDITS50_for_Linux_2017Feb20.zip). It contains C++ source for a console demo app, called RunEdits50, and a make file for g++. Just set up a new project in VC++, import the source files, and build it. The app demonstrates many features of EDITS50, including running edits, generating feedback to an Html page. (Note: you will probably want to update the variables at the top of function main; this sample was written back when the current metafile was at v15.)

    in reply to: Need to build an interface for XML to SQL Datadata #9688

    There is a problem with memory not being cleaned out

    You need to run a profiler to pin down exactly where your trouble is, but just guessing it seems you are allocating memory for repetitive processing and counting on the C# garbage collection process to take care of things. Even as a C++ programmer I learned I have to “think in C” when writing batch processing software. (I love the Standard Template Library, but it’s not the right answer when nanoseconds count.)

    So look for ways to re-use allocated objects. For instance, in the C++ code sample for running edits, the XMLPlus API shows that I build a list of allocated objects describing each of the data items I’ll be processing, building it just once, outside the processing loop. Most of the properties of those objects are unchanging (name, length, ParentXMLElement, etc.), but for running edits the value property needs to change for each case. So for each data record, I simply update the data value. Try running edits on large data files (I had a test file of more than 83,000 cases… and boy, did that make a huge XML file! … performance did not degrade from the first case to the last).

    Anyway, I recommend using a good profiler. I always relied upon AQTime from SmartBear. I used it for C++ and Object Pascal projects, but it also supports the .NET languages.

    in reply to: Need to build an interface for XML to SQL Datadata #9667

    In your code, where and how is readitem_callback declared?

    Is its value assigned in the callback function you passed to XMLPlus_OpenXmlDataFile? That is the function I want to see. From the context of what you wrote above, I’m guessing readitem_callback is a global variable. Its scope will be important to this discussion.

    I don’t mean to be haranguing you, it’s just that resolving this issue in this public forum could benefit others. I don’t want to leave the questions unanswered or incomplete.

    Thanks, Jeff.

    Kathleen

    in reply to: Need to build an interface for XML to SQL Datadata #9663

    The callback parameters are the same regardless of whether you are processing by naaccrNum or naaccrId and have the same format. Maybe I’m wrong here, but when I look at this function from my help file, isn’t parameter “value” a C# string? Or are you having to do something to it when you use this code by naaccrNum?

    private static void ReadItemByEitherNaaccrIdOrNaaccrNum(
        System.IntPtr owner,
        int patint_ordinal,
        int tumor_ordinal,
        [InAttribute()][MarshalAsAttribute(UnmanagedType.LPStr) string NaaccrId,
        int NaaccrNum,
        [InAttribute()][MarshalAsAttribute(UnmanagedType.LPStr) string value)
    {
        // etc.
    }

    The ByNaccrId call is different in that I need to use the pointer that is returned. The process of building a char array and a stringbuilder routine to build the value is what I was hoping there was some examples of since I am having trouble getting the pointer to the value working.

    I interpret the above syntax to be an instruction to C# that the parameter will arrive as a pointer to array of char, which C# converts on the spot to a C# string. You shouldn’t have to do anything else special to be able to use these strings in your code.

    May I see the code for the callback function in your code that works when you are retrieving data by value?

    Kathleen, still hoping a C#-to-C maven will join this thread!

    in reply to: Need to build an interface for XML to SQL Datadata #9661

    Jeff, I’m getting confused.

    I thought you were working with the functions to read the data file, but the declarations you just quoted are for reading the dictionary.

    When reading the data file, you pass the callback parameters (owner, callback_func) to the XMLPlus_OpenXmlDataFile function. My thinking was that you’ll be calling the “XMLPlus_GetItemDataBy[NaaccrId or NaaccrNum]” function gazillions of times, so just post the callback once for the entire run. (There is some set-up involved with posting a callback in the DLL, not costly, but why waste cycles? Reasonable people can argue about my design choices…)

    Looks like I have the problem using a pointer return variable to access the actual string.

    The “GetItemDataBy” functions don’t retrieve the data value directly; instead, the DLL locates the value by the identifier, and then sends the “answer” back through the callback function. I have a very good reason for this design choice: If you expected the DLL to populate a reference variable supplied in your direct call, you would have to first ask how long that string value is so that you could size your string appropriately (otherwise, danger of buffer overflow!). But through the magic of pointers, the DLL is maintaining the contents of the requested item in its local memory long enough to stream it to your callback function, where you can capture it to a C# string and manage that memory yourself.

    Take another look at the C++ sample, “Run EDITS on NAACCR XML Data File”, and look for XmlToFlatReadItemByNaaccrNum and XmlToFlatReadItemByNaaccrId. Particularly for preparing the EDITS buffer dynamically, you can see that more steps are required when you have to do the look-up by naaccrId, but it can be done.

    Kathleen

    in reply to: Need to build an interface for XML to SQL Datadata #9657

    Jeff,

    Would you mind running a test for me? You said you were able to get your code to retrieve data values using the “byNaaccrNum” function. Could you run the loop so that you first call “byNaaccrNum” and then call “byNaaccrId”? I’m trying to narrow the problem down to support my hunch about the declarations, and need to know that all of the other functions are working as advertised.

    So the pseudo-code would look like this (note that all of these calls should capture and test the return value… this is just pseudo-code for program flow):

    XMLPlus_Initialize(...)
    XMLPlus_OpenXmlDataFile(...)
    
    /* while not end-of-file */
    {
        XMLPlus_ReadNextPatient(...)
    
        XMLPlus_GetPatientTumorsCount(...)
    
        for (<iterate over tumors for current patient>)
        {
            XMLPlus_ReadTumor(...)
    
            for (<iterate list of data items you want>)
            {
                 XMLPlus_GetItemDataByNaaccrNum(...)
                 XMLPlus_GetItemDataByNaaccrId(...)
            }
        }
    }
    
    XMLPlus_CloseXmlDataFile(...)
    XMLPlus_Exit(...)

    You might even hard-code some items for this test, e.g., naaccrNum=390 and naaccrId=”dateOfDiagnosis”. Let me know what happens.

    Thanks,
    Kathleen

    in reply to: Need to build an interface for XML to SQL Datadata #9636

    Perhaps the byNaccrrid function will not work in C#?

    Oh, it will work all right. I just have to enlist the help of a C# programmer at the CDC to look at it. Dollars to donuts it will be the something about the declarations.

    The help file says the C# declaration for this function should look something like this:

    [DllImportAttribute("XMLPlus.dll", EntryPoint = "XMLPlus_GetItemDataByNaaccrId")]
    public static extern int XMLPlus_GetItemDataByNaaccrId(int XmlId,
       [InAttribute()] [MarshalAsAttribute(UnmanagedType.LPStr)] string naaccrId);

    (I just typed that from the help file, so please look to it there.) My hunch is that your declaration is not properly converting a C# string to a pointer to array of char, which of course is what you need for a C interface across languages. In the meantime, I’ll see if I can get somebody else to write a test for these functions at the CDC.

    Kathleen

    in reply to: Need to build an interface for XML to SQL Datadata #7580

    Hi Jeff,

    Cant seem to get the XMLPLUS DLL function: XMLPlus_GetItemDataByNaaccrId(const int XmlId,const char* naaccrId) working

    First, what are the return values of your calls into these functions? If any is anything except the all-is-well 0 (zero), you should use the functions in the chapter Handling System Errors to get better information.

    Let’s start with that, and if needed we can dig into your code. Please note, however, that my language is C++. If we end up in some technical area of C#, we’re going to need to enlist the help of somebody with expertise in that area.

    Is it reasonable to require the NAACCR Number in the data?

    FWIW, XMLPlus.dll always writes the data including both the naaccrId and the naaccrNum, because it makes editing the data more efficient. As to being able to require the naaccrNum, that probably depends upon how much clout you have with your reporting entities. Maybe you can make it a rule for your region?

    Kathleen

    Hi Jeff,

    My view is similar, each tumor instance needs a check, so if the header patient info fails the edit should skip tumor data and skip to the next patient, is that something that is going to be in EDITS50?

    Yes, this is in EDITS50 (for what it’s worth, it was in EDITS40, too). As you’ll see when you start working with the EDITS50 API, your application will be notified of edit errors as they occur through the callback mechanism. So a decision about how to handle the discovery of an edit error being raised is in your control.

    Caveat: The Engine will run all of the edits in an edit set for the currently-loaded case record. If you catch an error upon running the Patient edit set, the Engine will run the remaining edits in that edit set and then return control to your processing loop. You may take whatever action you deem appropriate at that time… bail on that Patient, continue on to find out whether/how many Tumor level errors may exist, whatever.

    Kathleen

    in reply to: Need to build an interface for XML to SQL Datadata #7459

    Regarding this “Next Queued task”:

    * Integration of the CDC’s EDIT50.dll for scoring and reporting.

    If you are just starting with the EDITS50 API, you might want to invest a half hour watching the “Tour of the EDITS50 API documentation” WebEx (see https://staging.naaccr.org/forums/topic/edits50-tools-implementation-forum/ for the link).

    Kathleen

Viewing 15 posts - 1 through 15 (of 28 total)

Copyright © 2018 NAACCR, Inc. All Rights Reserved | naaccr-swoosh-only See NAACCR Partners and Sponsors