EN JP CN

Import external defects with kwdefectimport

In this topic:
Working with source defect files Create a taxonomy Create your .pconf file Create a defect parser Sample parser Import the defects Run kwdefectimport on the command line Running kwdefectimport as part of an automated build

Import external defects with kwdefectimport

You can use Kwdefectimport to import a list of external defects into Klocwork. These are defects that were discovered by an external tool or process, independent of Klocwork.

You run Kwdefectimport on the command line at build time after Kwbuildproject and before kwadmin -load. When run, it uses the data in the artifacts (tables and files) that are created by kwbuildproject, then creates new problem data for kwadmin to load.

Importing external defects into Klocwork involves the following steps:

Creating or obtaining a listing of defects. Depending on your implementation, this list can be any format such as a file, a stream, or a port.
Creating a .pconf checker configuration file onto which the defects are mapped to checkers.
Creating a defect parser that implements the interface com.klocwork.defectimport.IDefectParser to parse the defect listing file.
Running kwdefectimport as part of your analysis, after kwbuildproject but before kwadmin -load.

Working with source defect files

While the exact format of your defect source can vary, the data must be valid and it must contain at least the following four elements per defect:

Element	Description
Defect code	A string representing the high-level category of the defect. In the example below, the defect code is represented by the element `<ErrorId>`.
File name	The absolute path and file name or a relative path and file name of the source file that contains the current defect. If you specify the --working-directory argument to `kwdefectimport`, the file name is considered a relative link (relative to the value specified in the argument). If you don't specify a working directory, the file name must be an absolute path. In the example below, the file name is represented by the `F` attribute.
Line number	The line number (in the source file) that contains the defect. In the example below, the file name is represented by the `L` attribute.
Message	Optional. The message explaining the defect. The message is displayed within Klocwork and provides context to the user.

The following listing shows a typical defect source file.

<ErrorId Id="ERR_TYP_TIMER_NOT_ACTIVE" NumberOfEntries="4">
  <Msg F="czagenmx.sdl" L="2786" S="C">timer t_wf_czg not activated in any path</Msg>
  <Msg F="czahanmx.sdl" L="5051" S="C">timer t_wf_czg_sup_ack not activated in any path</Msg>
  <Msg F="czahanmx.sdl" L="5074" S="C">timer t_wf_czg_sup_ack not activated in any path</Msg>
  <Msg F="czahanmx.sdl" L="5092" S="C">timer t_wf_czg_sup_ack not activated in any path</Msg>
</ErrorId>

Note: The sample also contains the S attribute which represents the severity level of each defect. Since the severity level of each defect is defined in the taxonomy, this attribute can be ignored by the parser.

Create a taxonomy

Before you can import defects, you must define the defect taxonomies, categories, and defect types with which the external defects will be associated. This is done by creating a .pconf file, which is an XML file that defines the categories and severity types of the defect, and contains the error message that is displayed to the user.

The following example shows a snippet from a .pconf file.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<checkers version="1.4">
    <severitytable max="10" warningStart="3" locale="en">
        <severity name="Critical" number="1"/>
         ...
    </severitytable>
    <categories name="TN SDL SDT" locale="en">
        <category name="Err.Lex.Tn">
            <error id="ERR_LEX_COM_IN_COM"/>
            <error id="ERR_LEX_LONG_ID"/>
			<error id="ERR_LEX_RESERVED"/>
        </category>  
        ...
    </categories>
    <checkergroup api="none" id="TN.DEFAULT.CHECKERGROUP" language="SDL/SDT">
        <checker id="ERR.LEX.TN">
            <error autogroupby="0" enabled="true" id="ERR_LEX_LONG_ID" severity="3">
                <title locale="en" text="ID longer than 31 characters"/>
                <title locale="ja" text="ID longer than 31 characters"/>
                <message locale="en" text="{0}"/>
                <message locale="ja" text="{0}"/>
            </error>
            <error autogroupby="0" enabled="true" id="ERR_LEX_RESERVED" severity="3">
                <title locale="en" text="ID is reserved in ANSI C"/>
                <title locale="ja" text="ID is reserved in ANSI C"/>
                <message locale="en" text="{0}"/>
                <message locale="ja" text="{0}"/>
            </error>
            <error autogroupby="0" enabled="true" id="ERR_LEX_COM_IN_COM" severity="3">
                <title locale="en" text="Missing close comment"/>
                <title locale="ja" text="Missing close comment"/>
                <message locale="en" text="{0}"/>
                <message locale="ja" text="{0}"/>
            </error>			
        </checker> 
        ...           
    </checkergroup>
</checkers>

The .pconf file contains three child elements that you must define and configure:

Element	Description
<severitytable>	Defines the severity hierarchy.
<categories>	Defines a set of checker categories, where each category contains a set of related checkers. Typically categories are used to organize checkers by function. For example, "Precision loss" or "Dangerous Type casts". These are typically user-defined.
<checkergroup>	Defines a group of related checkers. For example, "builtin C# path checkers". These are typically user-defined.

Create your .pconf file

Follow the walk-through below to create and import your .pconf file.

Create an XML file to contain the checker configuration information. You can start with an existing configuration file and a text editor, or you can use the Klocwork configuration editor to create your .pconf file.
For more information about creating and modifying a taxonomy, see Configuring checkers for the integration build analysis.

Add the <severitytable> element to your .pconf file. This is the first child of the root element. You can leave this section as is since the severity levels may be arbitrary depending on your workflow.

<severitytable max="10" warningStart="3" locale="en">
  <severity name="Critical" number="1"/>
  <severity name="Error" number="2"/>
  <severity name="Warning" number="3"/>
  <severity name="Review" number="4"/>
  <severity name="Severity 5" number="5"/>
  <severity name="Severity 6" number="6"/>
  <severity name="Severity 7" number="7"/>
  <severity name="Severity 8" number="8"/>
  <severity name="Severity 9" number="9"/>
  <severity name="Severity 10" number="10"/>
</severitytable>

Add the <categories> element to your .pconf file:
```
<categories name="TN SDL SDT" locale="en">  
  <category name="Err.Sym.Tn">
    <error id="ERR_SYM_NO_TERMINATOR"/>
    <error id="ERR_SYM_EXP_UNCHANGED"/>
    <error id="ERR_SYM_ENUM_LACK_ANSWER"/>
    <error id="ERR_SYM_UNREACHABLE_STATEMENT"/>
  </category>
</categories>   
```
- The <categories> element contains <category> elements that define the actual checker categories. The name attribute of both the <categories> and <category> element and is user-defined. You should specify a name for the <category> that represents the defect types it will contain.
- The id attribute of the <error> element is a direct mapping to the <ErrorID> specified in the external defect listing.

Add the <checkergroup> element to your .pconf file:

<checkergroup api="none" id="TN.DEFAULT.CHECKERGROUP" language="SDL/SDT">
  <checker id="ERR.LEX.TN">
   <error autogroupby="0" enabled="true" id="ERR_LEX_LONG_ID" severity="3">
     <title locale="en" text="ID longer than 31 characters"/>
     <title locale="ja" text="ID longer than 31 characters"/>
     <message locale="en" text="{0}"/>
     <message locale="ja" text="{0}"/>
   </error>
   <error autogroupby="0" enabled="true" id="ERR_LEX_RESERVED" severity="3">
     <title locale="en" text="ID is reserved in ANSI C"/>
     <title locale="ja" text="ID is reserved in ANSI C"/>
     <message locale="en" text="{0}"/>
     <message locale="ja" text="{0}"/>
    </error>
  </checker>

The elements in <checkergroup> are described in the tables below:

Element	Attributes
<checkergroup>	api specify "none" to indicate that no checker should be executed. id user-defined language user-defined, typically represents the programming language of the source code.
<checker>	id user-defined, but typically represents the name of the group of defect types it encapsulates.
<error>	autogroupby set to 0. enabled set to true. id the id attribute of the <error> element is a direct mapping to the <ErrorID> specified in the external defect listing. severity a value based on the interpretation of the severity levels defined in the defects listing file.
<title>	locale the locale code of the title string. For example, "en" or "ja". text a string that provides a simple overview of the defect.
<message>	locale the locale code of the title string. For example, "en" or "ja". text A string that provides a simple overview of the defect or `{0}` to pass the message specified in the defect listing. Since the defect messages have already been generated and the objective is just to associate the text of the message with lines in source files, the message is simply passed as a variable.

Save your XML file and copy it to the /plugins directory in your Klocwork install path.

Create a defect parser

To import defects into Klocwork, you must write a Java class that implements the interface com.klocwork.defectimport.IDefectParser. In addition, this implementation must have a public single argument constructor that accepts a String. This String represents the resource that contains the listing of external defects to be imported into Klocwork. This resource can be a file, a stream, a port or any other user-defined source. In the example below, it represents the full path to the file that contains the defects. Note also that there is no public method that can be explicitly called from within Kwdefectimport that will create an Iterator to satisfy the following interface method:

Iterator<SourceDefect> getDefects();

Therefore, the parser must prepare this Iterator from within its constructor. In the example below, see the call to the implementation-specific method, parse(), from within the constructor.

Sample parser

In this complete example of a parser, the defects are exported as XML. The implementation of IDefectParser extends SAX's DefaultHandler:

package com.thirdparty.defectexport.parser;

import com.klocwork.defectimport.IDefectParser;
import com.klocwork.defectimport.SourceDefect;
import org.jetbrains.annotations.NonNls;
import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;

import javax.xml.parsers.ParserConfigurationException;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
import java.io.File;
import java.io.IOException;
import java.util.*;

public class ThirdPartyKlocworkDefectParser extends DefaultHandler implements IDefectParser {
    //where the listing of the source defects resides (file, stream?, port?, etc)
    private final String defectsSource;
    //list of items discovered at the defectsSource, each of which having been converted to a SourceDefect
    private final List<SourceDefect> sourceDefects = new ArrayList<SourceDefect>();

    private SourceDefect sourceDefect;
    private boolean inByErrorId;
    private boolean inMsg;
    private String currentErrorId = "";
    StringBuilder buffer = new StringBuilder(1024);

    public ThirdPartyKlocworkDefectParser (final String defectsSource) throws ParserConfigurationException, SAXException, IOException {
        this.defectsSource = defectsSource;
        parse();
    }

    @Override
    public void startElement(final String uri, final String localName, final String qName, final Attributes atts) throws SAXException
    {
        buffer.setLength(0);

        if(localName.equals(ELEM_BY_ERROR_ID)){
            inByErrorId = true;
        }

        if(inByErrorId && localName.equals(ELEM_ERROR_ID)){
            final String attributeId = atts.getValue(ATTR_ID);
            if(!attributeId.equals(currentErrorId)){
                currentErrorId = attributeId;
            }
        }

        if(inByErrorId && localName.equals(ELEM_MSG)){
            inMsg = true;
            sourceDefect = new SourceDefect();
            sourceDefect.setErrorId(currentErrorId);
            final String fileName = atts.getValue(ATTR_FILE);
            sourceDefect.setFileName(fileName);
            sourceDefect.setLineNumber(Integer.valueOf(atts.getValue(ATTR_LINE)));
            sourceDefect.setMessage(qName);
        }
    }

    @Override
    public void endElement (final String uri, final String localName, final String qName)
            throws SAXException
    {
        if(inByErrorId && localName.equals(ELEM_MSG)){
            sourceDefect.setMessage(buffer.toString());
            sourceDefects.add(sourceDefect);
            inMsg = false;
        }
    }

    @Override
    public void characters(final char[] ch, final int start, final int length) throws SAXException {
        if(inByErrorId && inMsg){
            buffer.append(ch, start, length);
        }
    }

    @Override
    public Iterator<SourceDefect> getDefects() {

        return Collections.unmodifiableCollection(sourceDefects).iterator();
    }

    private void parse() throws ParserConfigurationException, SAXException, IOException {
        final SAXParserFactory spf = SAXParserFactory.newInstance();
        spf.setNamespaceAware(true);

        final SAXParser parser = spf.newSAXParser();
        final XMLReader reader = parser.getXMLReader();
        reader.setContentHandler(this);
        reader.parse(new File(defectsSource).toURI().toURL().toString());
    }

    @Override
    public void close() {
        //not required for this implementation
    }

    @SuppressWarnings("HardCodedStringLiteral")
    @NonNls
    private static final String ELEM_BY_ERROR_ID = "ByErrorId";

    @SuppressWarnings("HardCodedStringLiteral")
    @NonNls
    private static final String ELEM_ERROR_ID = "ErrorId";

    @SuppressWarnings("HardCodedStringLiteral")
    @NonNls
    private static final String ATTR_ID  = "Id";

    @SuppressWarnings("HardCodedStringLiteral")
    @NonNls
    private static final String ELEM_MSG = "Msg";

    @SuppressWarnings("HardCodedStringLiteral")
    @NonNls
    private static final String ATTR_FILE  = "F";

    @SuppressWarnings("HardCodedStringLiteral")
    @NonNls
    private static final String ATTR_LINE  = "L";

}

Import the defects

Once you've defined your external defects file, you've mapped those defects to a checker configuration, and you've written your parser, you can run kwdefectimport to import the external defects into Klocwork. You run Kwdefectimport from the command line, after running Kwbuildproject, but before running kwadmin -load:

At runtime, kwdefectimport opens the source defects listing file, reads the defect types and maps them to the appropriate source file and line of code. In the tables directory the files file.dat, and entity.dat are modified. Defects are written to a new file called problem_ext.pbf which is then loaded by kwadmin -load.

Run kwdefectimport on the command line

The following example shows kwdefectimport run from the command line:

kwdefectimport --tables-directory "C:/Klocwork/Server XY.Z/projects_root/projects/abc_main" 
--working-directory "C:/_workspaces/ABC_Main/src" --parser-class com.klocwork.defectimport.parser.TNSDLDefectParser 
--rejected-defects C:/tables/rejecteddefects.json --ext-libs "C:/Klocwork/lib/kwdefectimport.jar" 
--ext-libs "C:/Klocwork/lib/another_required_library.jar" "C:\already_analyzed_defects_in_proprietary_format\defects_01.xml" "C:\already_analyzed_defects_in_proprietary_format\defects_02.xml

For more information about running kwdefectimport and for a complete description of the options, see Kwdefectimport.

Running kwdefectimport as part of an automated build

If you include kwdefectimport as a regular part of your automated Klocwork tool chain (between kwbuildproject and kwadmin -load), you will require kwdefectimport's exit code to determine whether to continue. See Kwdefectimport for a complete list of error codes.

The following example uses a DOS batch script to demonstrate how to use the exit code to determine the action to take based on the exit code. In this example, there are 2 batch files: caller.bat and run_kwdefectimport.bat The file caller.bat is responsible for executing kwdefectimport and returning its exit code to the caller. The file run_kwdefectimport.bat calls caller.bat and then makes execution decisions based on thecaller.bat's exit code.

run_kwdefectimport.bat

@echo off
set errorlevel=
"C:\Klocwork\Server 12.0\bin\kwdefectimport.exe" --tables-directory C:\tables\20131024 --working-directory "C:/_workspaces/Main/REP/sources/kwdefectimport/test_resources/source_dir" --parser-class com.klocwork.defectimport.parser.TNSDLDefectParser --rejected-defects C:/tables/rejecteddefects.json "C:/_workspaces/Main/REP/sources/kwdefectimport/test_resources/sdl_defects/tncheck.xml"
exit /b %errorlevel%

caller.bat

caller.bat
@echo off
call run_kwdefectimport.bat
echo Exit Code is %errorlevel%
set atemp=%errorlevel%
echo atemp is %atemp%

set BUILD_FAILED_EXIT_CODE=1
set DEFECT_TYPES_NOT_REGISTERED_EXIT_CODE=16
set DUPLICATE_DEFECTS_FOUND_EXIT_CODE=32
set SOURCE_FILES_NOT_FOUND_EXIT_CODE=64
set SOURCE_FILES_NOT_REFERENCED_EXIT_CODE=128

set /A critical="%atemp%&%BUILD_FAILED_EXIT_CODE%"
set /A dnr="%atemp%&%DEFECT_TYPES_NOT_REGISTERED_EXIT_CODE%"
set /A dup="%atemp%&%DUPLICATE_DEFECTS_FOUND_EXIT_CODE%"
set /A snf="%atemp%&%SOURCE_FILES_NOT_FOUND_EXIT_CODE%"
set /A snr="%atemp%&%SOURCE_FILES_NOT_REFERENCED_EXIT_CODE%"

echo dnr %dnr%
echo dup %dup%
echo snf %snf%
echo snr %snr%

if "%dnr%" == "%DEFECT_TYPES_NOT_REGISTERED_EXIT_CODE%" echo Defect type(s) not registered
rem if "%dnr%" == 16 goto whatever_you_want_to_do_with_this_case
if "%dup%" == "%DUPLICATE_DEFECTS_FOUND_EXIT_CODE%" echo Duplicate defect(s) found
rem as above
if "%snf%" == "%SOURCE_FILES_NOT_FOUND_EXIT_CODE%" echo Source file(s) not found
rem as above
if "%snr%" == "%SOURCE_FILES_NOT_REFERENCED_EXIT_CODE%" echo Source file(s) not referenced
rem as above
if "%critical%" == "1" echo BUILD FAILED
rem You probably don't want to continue
REM place actions to take here

pause
rem Pause would _not_ be used in a production environment