Determining the Idle Power of an Intel® Xeon Phi™ Coprocessor

Abstract

This document gives platform designers, thermal engineers, hardware engineers, and computer architects instructions on how to acquire idle power readings from the Intel® Xeon Phi™ coprocessor.

There are two access methods by which the server management and control panel component may obtain status information from the Intel® Xeon Phi™ coprocessor. The “in-band” method utilizes the Symmetric Communications Interface (SCIF), the capabilities designed into the coprocessor OS, and the host driver to deliver the Intel® Xeon Phi™ coprocessor status. It also provides a limited ability to set specific parameters that control hardware behavior. An example application using this interface would be ‘micsmc’, which is provided with the Intel® Manycore Platform Software Stack (Intel® MPSS), which reads the power in-band.

The same information can be obtained using an “out-of-band” method. This method starts with the same capabilities in the coprocessor OS, but sends the information to the System Management Controller (SMC) using a proprietary protocol. With this method, the coprocessor idle power measurements can be made without waking up the card.

The Intel® Xeon Phi™ coprocessor communicates with the baseboard management controller (BMC) or peripheral control hub (PCH) over the System Management Bus (SMBus) using the standard Intelligent Platform Management Bus (IPMB) protocol.  The SMC responds to queries from the platform’s BMC using the Intelligent Platform Management Interface (IPMI).  Through the Inter-Integrated Circuit (I2C) interface, the SMC can communicate with the Intel® Xeon Phi™ coprocessor and to sensors located on the PCIe card.

Image may be NSFW.
Clik here to view.Figure 1: Intel® Xeon Phi™ Coprocessor Board Schematic

For Intel® Xeon® processor E5-2600 v3 product family platforms, the Intel® ME can provide access to the SMC on the Intel® Xeon Phi™ coprocessor with very little effort, and can read the idle power from the SMC.

For Intel® Xeon® processor E5 V2 family platforms, the Intel® ME does not have a mechanism for reading power from the Intel® Xeon Phi™ coprocessor and so relies on the BMC to provide sensor information.  This however, requires that the BMC implements some mechanism for communicating with the Intel® Xeon Phi™ coprocessor either via special OEM commands or through a bridging mechanism.

Image may be NSFW.
Clik here to view.Figure 2: Example of a topology where the BMC is connected to Intel® Xeon Phi™ coprocessors

Unlike the sensors that can be accessed via the BMC’s SDR, the sensors of the Intel® Xeon Phi™ coprocessor are abstracted behind a different I2C bus.  In order to access these sensors, the user needs to be familiar with the I2C network diagram and the mechanism for accessing the bus.  Also, they might also need to be exposed via special BMC OEM commands or via a third-party vendor’s help.  You can find out more details about how to do this from the Intel® Xeon Phi™ Coprocessor Datasheet.

The example scripts were tested on both an Intel® Xeon® processor E7 V2 server product with four 7120A Intel® Xeon Phi™ Coprocessors, and an Intel® Xeon® processor E5-2600 v3 server product with two 7120A Intel® Xeon Phi™ Coprocessors.

On these platforms, the BMC has implemented several OEM commands that provide reverse PCIe SMBus proxy.  On the Intel® Xeon® processor E5 V2 family* platforms, the format is below:

Table 1: Get MIC Card Info Command (30h E3h)

* Note that this changed to match the E8h command in later versions of the document. Also, the Intel Manufacturer ID – 000157h is removed from the request and response parts of the command below.

On the Intel® Xeon® processor E5-2600 v3 server products family and Intel® Xeon® processor E7 V2 family, the format is slightly different:

Table 2: Get MIC Card Info Command (30h E8h)

The first step that needs to get determined is to find out how many Intel® Xeon Phi™ coprocessors are in the system.  Then once this is done, the bus number, slot number, and slave address of each Intel® Xeon Phi™ coprocessor needs to be determined.  The bus number determines which SMBus interface in which the Intel® Xeon Phi™ coprocessor was detected.  The slot number identifies which PCIe slot that the Intel® Xeon Phi™ coprocessor is inserted into.  Finally, the slave address is the I2C slave address of the SMC on the Intel® Xeon Phi™ coprocessor. With this information, commands can be sent directly to the Intel® Xeon Phi™ Coprocessor according to the commands in the Intel® Xeon Phi™ Coprocessor Datasheet, section 6.6.3.

Perl is a great programming language that is great for scripting and it can be used to automate complicated IPMI commands using IPMItool.  In the example Perl subroutines below, IPMItool is used to send PCIe slot commands and determine how many cards are on the system:

sub Read_PCIe_smbus_slot_card_info {
    my ($count_KNC) = @_;
    my $str0 = "ipmitool raw 0x30 0x".$eX_cmd." 0x00";
    #printf ("PCIe SMbus slot card info request: $str0\n");
    my $str1 = `$str0`;
    #If the response is "Unable..." this means that the command isn't implemented
    if (substr($str1,1,1) eq "") {
        #Let's see if this is an Intel(R) Xeon(R) processor E7 V2 product or an
        #Intel(R) Xeon(R) processor E5-2600 v3 product
        $eX_cmd= "e8";
        $str0 = "ipmitool raw 0x30 0x".$eX_cmd." 0x00";
        #printf ("PCIe SMbus slot card info request: $str0\n");
        $str1 = `$str0`;
    }
    if (substr($str1,1,1) eq "") {
        die("\nThe BMC on your platform does not support the PCIe Slot SMBus Slot Command. Please"," consult with your BMC vendor. This program will now quit.\n");
    }
        $count_KNC= substr($str1,1,3);
    #printf ("$count_KNC\n");

    return $count_KNC;
}

Next the Intel® Xeon Phi™ coprocessor’s addressing parameters can be determined with the following command:

sub Read_PCIe_smbus_slot_card {
    my ($key) = @_;
    my $str0 = "ipmitool raw 0x30 0x".$eX_cmd."".$key;
    #printf "$str0\n";
    my $str1 = `$str0`;
    #printf "$str1\n";
    my $bus_num= substr($str1,4, 2);
    my $slot_num= substr($str1,13,2);
    my $slave_address= substr($str1,16,2);
    return $bus_num, $slot_num, $slave_address;
}

Now with the communication parameters of each Intel® Xeon Phi™ coprocessor, the BMC needs to provide a way to send the command to the SMC itself.  On the Intel® platforms that were just mentioned, this can be done using the Slot IPMB command below:

Table 3: Slot IPMB Command (3Eh 51h)

Here a command is sent to the card in this way:

Request:

Image may be NSFW.
Clik here to view.

Response (see below for the explanation):

               (00) 00 00 00 01 16 02 0f 57 01 00 60 00 d6 13 00 00

The response is broken down below.  For more details, see Table 4 after the explanation.

The byte in parentheses will not be shown in the response.  It is a successful completion code from the command 0x3E 0x51 which is not displayed by IPMItool.

The first byte represents the completion code 00h from the execution of the bridged Get Device ID command which normally would not be shown unless there is an error. It is displayed here because the Slot IPMB command simply returns the full content of the response without parsing the completion code. Byte 2 is the device ID (00h for unspecified). Byte 3 is the device revision (00h in this case and also indicating that the device does not provide device SDRs).

Byte 4 refers to the Firmware Revision 1 (01h indicates Major Firmware Revision of 1, and indicating normal operation). Byte 5 refers to the Firmware Revision 2 (16h indicates 1.6).  (These two bytes combined together would correspond to the SMC FW’s revision, in the case, is 1.16).

Byte 6 refers to the IPMI version (02h indicates 2.0).  Byte 7 represents Additional Device Support (0Fh means that the device supports a FRU, SEL, SDR, and sensor devices). Bytes 8 – 10 are the manufacturer ID, LS byte first (000157h means Intel’s manufacturer ID).

Bytes 11-12 are the Product ID, LS byte first (0060h). Bytes 13-16 stand for the auxiliary firmware revision information (D6130000h).

Table 4: Get Device ID Command

Image may be NSFW.
Clik here to view.

The command above is a simple “Get Device ID” command which is common among most BMC and other IPMI devices.  Other commands can be sent to the SMC in a similar way.

The sensor names in the SMC’s SDR are static and do not change from release to release, however, the sensor numbers are not always static.  These numbers may change in future releases, so it is a good idea to query the BMC each time the BMC or SMC firmware has changed.  Doing this requires a few steps and the construction of some simple subroutines.  The “for” loop below shows how to send multiple IPMI commands in order to get some basic information out of the card:

for (1)
{
    #e3 is the PCIe SMBus Slot card command for the Intel(R) Xeon(R) processor E5-4600 v2 product family platforms;
    #e8 is for Intel(R) Xeon(R) processor E5-2600 v3 product family and Intel(R) Xeon(R) processor E7 V2 family
    #my $eX_cmd = "e3";
    my ($count_KNC1) = Read_PCIe_smbus_slot_card_info($eX_cmd);
    #printf "$count_KNC1\n";
    my $count_KNC = substr($count_KNC1,1,3);
    printf "The number of Intel(R) Xeon Phi(TM) coprocessors is $count_KNC\n";

    printf("Which Intel(R) Xeon Phi(TM) coprocessor PCIe card would you like to query? (0..n) ");
    my $key = getc(STDIN);
    $key +=1;
    if ($key > $count_KNC) {
        $key = 1;
    }
    my $key2 = $key - 1;

    my ($bus_num, $slot_num, $slave_address) = Read_PCIe_smbus_slot_card($key);
    printf "\nFor Intel(R) Xeon Phi(TM) coprocessor PCIe card#$key2:\nThe Bus Number is 0x$bus_num, Slot Number is 0x$slot_num, Slave Address is 0x$slave_address\n";
    (my $count_SDR) = Read_SMC_SDR_Repository($bus_num, $slot_num, $slave_address);
    printf "\nThe SMC's SDR Repository has 0x".$count_SDR." records\n";
    my $count_SDR_dec = hex($count_SDR);

    for (my $i=0; $i < $count_SDR_dec; $i++ ){
        Scan_SMC_SDR_Repository_for_Idle_Power ($key, $i, $bus_num, $slot_num, $slave_address);
    }

}

The first command calls the “Read_PCIe_smbus_slot_card_info()” subroutine to get the number of Intel® Xeon Phi™ coprocessors, and then ask the user which card they want to read.  Then the “Read_PCIe_smbus_slot_card()” subroutine is called to get the Intel® Xeon Phi™ coprocessor’s bus number, slot number, and slave address.

Next the “Read_SMC_SDR_Repository()” subroutine is called to find the number of SDR records on the SMC:

sub Read_SMC_SDR_Repository {
    my ($bus_num, $slot_num, $slave_address)=@_;
    my $str0 = "ipmitool raw 0x3e 0x51 0x".$bus_num." 0x".$slot_num." 0x".$slave_address." 0x0a 0x20";
    #printf "$str0\n";
    my $str1 = `$str0`;
    #printf "$str1\n";
    my $count_SDR = substr($str1, 7,2);
}

Here is the structure of that command and the output (based on one particular Intel® Xeon Phi™ coprocessor):

Request:

ipmitool raw 0x3e 0x51 0x02 0x96 0x30 0x0a 0x20

Response:

 00 51 1c 00 00 00 00 00 00 00 00 00 00 00 01

The third byte returns the number of record present in the SDR.  In this example, the card has 1Ch records, or 28 records.

Next the “Scan_SMC_SDR_Repository_for_Idle_Power()” subroutine makes more IPMItool calls to read the sensor value.  A “for” loop calls this subroutine up to 28 times until the desired sensor is found, in this case, “avg_power1”.  This sensor is the sum of the three power sensors on the card and is averaged over time window 1, so it is a good indicator of the card’s power.

The subroutine below is broken down into parts:

sub Scan_SMC_SDR_Repository_for_Idle_Power {
    #my $SDR_no=@_[0];
    my ($key, $SDR_no, $bus_num, $slot_num, $slave_address)=@_;
    my $str_sdr = "ipmitool raw 0x3e 0x51 0x".$bus_num." 0x".$slot_num." 0x".$slave_address." 0x0a 0x23 0x00 0x00 ".$SDR_no." 0x00 0x07 0x0f";
    my $sensor = `$str_sdr`;
    #printf ("First part of the SDR $SDR_no is $sensor\n");
    my $sensor_no = substr($sensor,10, 2);

These first few commands involve reading the SDR record and finding out the contents from byte 07h until byte 0Fh.  There are different types of sensor data records, but the most common one is Type 01h, for a Full Sensor Record.  The first 8 bytes of the SDR description are shown below:

Table 5: Full Sensor Record - SDR Type 01h (First 8 Bytes)

Image may be NSFW.
Clik here to view.

Byte 8 gives the sensor number, which can be used to match it up with the sensor name.

    my $str0 = "ipmitool raw 0x3e 0x51 0x".$bus_num." 0x".$slot_num." 0x".$slave_address." 0x0a 0x23 0x00 0x00 ".$SDR_no." 0x00 0x2e 0xff";
    #printf "$str0\n";
    my $str1 = `$str0`;
    #printf ("Second part of the SDR $SDR_no is $str1\n");
    my $str2 = substr($str1, 15, length($str1));
    #printf "The name of the SDR is in ASCII here: $str2\n";
    $str2 =~ s/\s+//g;
    #printf "Remove spaces: $str2\n";
    my $str3= hex_to_ascii($str2);
    #printf "The sensor name of SDR#$SDR_no is '$str3' (Sensor# 0x$sensor_no)\n";
    #Other sensors can be substituted for "avg_power1" if it is desired to poll a different sensor */
    if ($str3 eq "avg_power1") {
        #printf("Entered if comparison\n");
        my $str4 = "ipmitool raw 0x3e 0x51 0x".$bus_num." 0x".$slot_num." 0x".$slave_address." 0x04 0x2d 0x".$sensor_no;

The above code reads the SDR record from bytes 2Eh until the end of the record. At byte 31h, or 49 in decimal, the name of the sensor is coded in ASCII character codes.  Here the bytes are saved into a variable, then converted from ASCII codes into characters, put into a string, and then compared to “avg_power1”.  If there is a match, then the right sensor is found.

Table 6: Full Sensor Record - SDR Type 01h (ID String Bytes)

Image may be NSFW.
Clik here to view.

The next step is to convert the raw data into something easily understood:

        # Get the M, B, Accuracy, Accuracy Exp, R exp, and B Exp for the SDR formula
        my $str6 = "ipmitool raw 0x3e 0x51 0x00 0x".$slot_num." 0x".$slave_address." 0x0a 0x23 0x00 0x00 ".$SDR_no." 0x00 0x18 0x06";
        my $str7 = `$str6`;
        # Only the M value seems to be needed for the formula. The other values can be ignored */
        my $M = substr($str7, 11, 2);
        #printf("M is $M\n");

The values for the ‘y=mx+b’ reading conversion are determined by reading bytes 25 - 30. M can be read from byte 25 and parts of 26.  Typically, in power sensors, only the M value is significant (reading the SDR at these bytes reveal 02h, which means to multiple the decimal value from the sensor by 2).

Table 7: Full Sensor Record - SDR Type 01h (M, B, Accuracy, R exp & B exp)

Image may be NSFW.
Clik here to view.

In the last several lines of the code below, these parameters are then used to send an IPMItool command to read the sensor:

          my $key2 = $key - 1;
        while (1){
            my $str5 = `$str4`;
            #printf ("The Sensor value of SDR#$SDR_no is $str5");
            printf ("Sensor '$str3' (0x$sensor_no) is $str5\n");
            my $str6 = substr($str5, 4, 2);
            #printf ("String6 is $str5\n");
            my $dec_num = hex($str6);
            #printf "dec_num is $dec_num\n";
            my $idle_pw = $dec_num * $M;
            printf("Intel(R) Xeon Phi(TM) coprocessor PCIe card#$key2:\nThe Bus number is 0x$bus_num, Slot Number is 0x$slot_num, Slave Address is 0x$slave_address: Power is $idle_pw W\n");
            sleep 1;
        }
    }
}

Request:

Image may be NSFW.
Clik here to view.

Response:                                               

 00 08 00 00

The first byte is the completion code, and 00 means that the command executed successfully.  The second byte is the raw sensor value.  The subroutine converts the value from hex to decimal, multiplies it by a factor of 2, and then prints the calculated value to the screen along with the Intel® Xeon Phi™ coprocessor’s number, bus number, slot number, and the slave address. To prevent the overloading of the bus, the subroutine waits approximately 1 second and then reads the sensor again.

There could be several factors keeping the Intel® Xeon Phi™ coprocessor idle power to be higher than expected. Here are some tips to reduce energy consumption:

• Intel® MPSS must be running in order to put the card in PC3 or PC6
• Power management is handled by Intel® MPSS and the coprocessor OS running on the card
• ‘micsmc’ will wake the card out of PC6, so micsmc must be shut down to allow the card to enter the PC6 idle state
• Shutting down the virtual interface on the host platform will prevent the card from being woken up by pings to the card.  Use the command “ifdown micN” where N represents the Intel® Xeon Phi™ Coprocessor number
• Always run the latest SMC firmware to make sure that your card supports power management (Note: Not all SKUs support all PC states)

There are some steps that need to be followed in order to get the sample Perl script described in this white paper.  Here are instructions how to do this on Red Hat*:

[intel]$ yum install perl

For SuSE*, use YaST in GUI mode.  From the command line, use “rug” if using SuSE* 10.1, or “zypper” if using 10.3. Please check SuSE* documentation for more details.

Once Perl is installed, enter the following command:

[intel]$ perl -MCPAN -e shell

Then at the new prompt:

cpan> install String::HexConvert ':all'

The Perl script and subroutines can be modified to read other sensors on the SMC if so desired.  Please check the Intel® Xeon Phi™ Coprocessor Datasheet for sensor names.  Also check the M, B, Tolerance, B, Accuracy, Accuracy exp, R exp, and B exp parameters from the SDR record when looking at other sensors.

Additional Resources

488073: Intel® Xeon Phi™ Coprocessor Datasheet, available from IBL/CDI

513973: Intel® Intelligent Power Node Manager 3.0 External Interface Specification using IPMI, Rev. 1.0.3, available from IBL/CDI

434090: Intel® Intelligent Power Node Manager 2.0 External Interface Specification Using IPMI, Revision 1.8, available from IBL/CDI

Intelligent Platform Management Interface Specification, Second Generation, v2.0, available publicly at http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-specifications.html

IPMItool Man page: http://linux.die.net/man/1/ipmitool

Todd Enger is a platform application engineer working overseas in Taipei, Taiwan for Intel Microelectronics Asia Ltd.  He specializes in software and firmware support of the Intel® Xeon Phi™ Coprocessor and is also working on enabling customers who will build platforms based upon the next generation Knights Landing processor. Todd has spent the last 10 years working in Taiwan, of which the past 4 years, he has been at Intel.  Prior to that, he worked for various OEMs and ODMs in the server, notebook, and smartphone areas.  Back in the US, Todd worked in the Chicago area until a business trip brought him to Taiwan.  After 2 weeks of astonishment, Todd used his ingenuity to find an opportunity in Taipei developing software on smartphones.  Todd received his BSE in Electrical and Computer Engineering from the University of Michigan-Dearborn.  In his spare time, he enjoys scuba diving in the waters around Taiwan, running, swimming, and hanging out at the beach.

Notices

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to:  http://www.intel.com/design/literature.htm

Intel, the Intel logo, VTune, Cilk and Xeon are trademarks of Intel Corporation in the U.S. and other countries.

*Other names and brands may be claimed as the property of others

Copyright© 2014 Intel Corporation. All rights reserved.

This sample source code is released under the Intel Sample Source Code License Agreement

General Notices:  Notices placed in all materials distributed or released by Intel

Benchmarking and Performance Disclaimers: Disclaimers for Intel materials that use benchmarks or make performance claims.

Technical Collateral Disclaimers: Disclaimers that should be included in Intel technical materials that describe the form, fit or function of Intel products.

Technology Notices: Notices for Intel materials when the benefits or features of a technology or program are described.  Note for technology disclaimers - if every product being discussed (e.g., ACER ULV) has the particular technology/feature, then you can remove the requirements statement in the disclaimer.  If you have multiple technical disclaimers, you can consolidate the "your performance may vary" statements and only put in a single "your mileage may vary".