Using Microsoft AI Phi-2 from FWH

[alerchster]

command prompt
Run vcvarsall.bat
Change to the directory from step 3 (temp).
Call there cmake .. (parameter is ..) and this will create the sln file for VS.

[Antonio Linares]

hi Antonio,

Could you kindly adapt this code to HMG ?

i´m not sure about "Callback" under HMG

will FWH CODE work when "LoadLibrary", "llama64.dll" or do i need DLL FUNCTION

Dear Jimmy,

Does HMG support DLL FUNCTION ? or similar ? Or should we use Harbour default DLL management functions ?

Can you convert the FWH GUI part of the example to HMG ?

many thanks

[Jimmy]

hi Antonio,

Does HMG support DLL FUNCTION ? or similar ? Or should we use Harbour default DLL management functions ?

Can you convert the FWH GUI part of the example to HMG ?

as i know there is no DLL FUNCTION under HMG only

HMG_CallDLL()
Calls a ANSI/Unicode DLL Function

HMG_CallDLL() detects the current code page and appropriately called the ANSI or Unicode version of the function.

Syntax:

HMG_CallDLL ( cLibName, [ nRetType ] , cFuncName, FuncArg1, ..., FuncArgN ) ---> xRetValue

nRetType: value type returned by the function (constants defined in hbdyn.ch)
- HB_DYN_CTYPE_DEFAULT
- HB_DYN_CTYPE_CHAR
- HB_DYN_CTYPE_CHAR_UNSIGNED
- HB_DYN_CTYPE_CHAR_PTR
- HB_DYN_CTYPE_CHAR_UNSIGNED_PTR
- HB_DYN_CTYPE_SHORT
- HB_DYN_CTYPE_SHORT_UNSIGNED
- HB_DYN_CTYPE_SHORT_PTR
- HB_DYN_CTYPE_SHORT_UNSIGNED_PTR
- HB_DYN_CTYPE_INT
- HB_DYN_CTYPE_INT_UNSIGNED
- HB_DYN_CTYPE_INT_PTR
- HB_DYN_CTYPE_INT_UNSIGNED_PTR
- HB_DYN_CTYPE_LONG
- HB_DYN_CTYPE_LONG_UNSIGNED
- HB_DYN_CTYPE_LONG_PTR
- HB_DYN_CTYPE_LONG_UNSIGNED_PTR
- HB_DYN_CTYPE_LLONG
- HB_DYN_CTYPE_LLONG_UNSIGNED
- HB_DYN_CTYPE_LLONG_PTR
- HB_DYN_CTYPE_LLONG_UNSIGNED_PTR
- HB_DYN_CTYPE_FLOAT
- HB_DYN_CTYPE_FLOAT_PTR
- HB_DYN_CTYPE_DOUBLE
- HB_DYN_CTYPE_DOUBLE_PTR
- HB_DYN_CTYPE_BOOL
- HB_DYN_CTYPE_BOOL_PTR
- HB_DYN_CTYPE_VOID
- HB_DYN_CTYPE_VOID_PTR
- HB_DYN_CTYPE_STRUCTURE
- HB_DYN_CTYPE_STRUCTURE_PTR

Source example:
cBuffer := SPACE (1024)
nCopyCharacters := HMG_CallDLL ("USER32.DLL", HB_DYN_CTYPE_INT, "GetWindowText", Win_1.HANDLE, @cBuffer, 512)
MsgInfo (cBuffer)

also ""Callback" Syntax is not available under HMG

---

i have made HMG CODE for GUI

Code: Select all  Expand view  RUN

#include "HMG.CH"
#include "hbdyn.ch"

MEMVAR _HMG_SYSDATA

PROCEDURE MAIN

DEFINE WINDOW AI_FORM   ;
   AT 0 , 0             ;
   WIDTH 722 HEIGHT 545 ;
   TITLE ""             ;
   ICON NIL             ;
   ON INIT Nil          ;
   ON RELEASE Nil       ;
   ON INTERACTIVECLOSE Nil ;
   ON SIZE Nil          ;
   ON MAXIMIZE Nil      ;
   ON MINIMIZE Nil      ;
   BACKCOLOR Nil

    DEFINE EDITBOX cPrompt
        ROW    10
        COL    20
        WIDTH  590
        HEIGHT 30
        VALUE PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
        FONTNAME "Arial"
        FONTSIZE 12
        TOOLTIP ""
        TABSTOP .T.
        VISIBLE .T.
        READONLY .F.
        HSCROLLBAR .T.
        VSCROLLBAR .T.
    END EDITBOX

    DEFINE EDITBOX cAnswer
        ROW    60
        COL    20
        WIDTH  590
        HEIGHT 420
        VALUE ""
        FONTNAME "Arial"
        FONTSIZE 12
        TOOLTIP ""
        TABSTOP .T.
        VISIBLE .T.
        READONLY .F.
        HSCROLLBAR .T.
        VSCROLLBAR .T.
    END EDITBOX

    DEFINE BUTTON oBtn
        ROW    10
        COL    620
        WIDTH  80
        HEIGHT 28
        ACTION DoAction()
        CAPTION "Start"
        FONTNAME "Arial"
        FONTSIZE 12
        TOOLTIP ""
        FLAT .F.
        TABSTOP .T.
        VISIBLE .T.
    END BUTTON

    DEFINE BUTTON oClear
        ROW    60
        COL    620
        WIDTH  80
        HEIGHT 28
        ACTION DoClear()
        CAPTION "Clear"
        FONTNAME "Arial"
        FONTSIZE 12
        TOOLTIP ""
        FLAT .F.
        TABSTOP .T.
        VISIBLE .T.
    END BUTTON

END WINDOW

   CENTER WINDOW AI_FORM
   ACTIVATE WINDOW AI_FORM

RETURN
 

but the Action, to call Llama, does not compile under HMG

ai_form.prg(94) Error E0022 Invalid lvalue '()'

Code: Select all  Expand view  RUN

PROCEDURE DoClear()
   AI_FORM.oAnswer.Value := ""
RETURN

PROCEDURE DoAction()
   AI_FORM.oBtn.VISIBLE := .F. // Disable()

   // does not compile under HMG
   Llama( "phi-2_Q4_K_M.gguf", RTrim( AI_FORM.cPrompt.Value ),;
   CALLBACK( { | cStr | AI_FORM.oAnswer.SetFocus(), AI_FORM.oAnswer.Value += cStr } )  )

   AI_FORM.oBtn.VISIBLE := .T. // Enable()
   AI_FORM.oBtn.SetFocus()

RETURN
 

have ask in HMG Forum for Solution

[Antonio Linares]

Dear Jimmy,

many thanks

Please try this using 64 bits only:

HMG_CallDLL( "llama64.dll", HB_DYN_CTYPE_VOID, "Llama", cModel, cPrompt, CallBack( { | cStr | AI_FORM.oAnswer.SetFocus(), AI_FORM.oAnswer.Value += cStr } ) )

and add this at the bottom:

Code: Select all  Expand view  RUN

#pragma BEGINDUMP

#include <hbapi.h>
#include <hbapiitm.h>

static PHB_ITEM pBlock;

static void callback( char * szMsg )
{
   PHB_ITEM pStr = hb_itemPutC( NULL, szMsg );
   hb_evalBlock1( pBlock, pStr );
   hb_itemRelease( pStr );
}

HB_FUNC( CALLBACK )
{
   pBlock = hb_gcGripGet( hb_param( 1, HB_IT_BLOCK ) );
   hb_retnll( ( HB_LONGLONG ) callback );
}

#pragma ENDDUMP

[Jimmy]

hi Antonio,

Please try this using 64 bits only:

Code: Select all  Expand view  RUN

HMG_CallDLL( "llama64.dll", HB_DYN_CTYPE_VOID, "Llama", cModel, cPrompt, CallBack( { | cStr | AI_FORM.oAnswer.SetFocus(), AI_FORM.oAnswer.Value += cStr } ) )
 

thx for Answer

have try that CODE but still get Error when compile

AI_FORM.prg(97) Error E0022 Invalid lvalue '()'

it seems me 3rd Parameter pFunc using Callback() Syntax fail under HMG

will continue to search Solution ...

[Jimmy]

hi Antonio,

Code: Select all  Expand view  RUN

HMG_CallDLL( "llama64.dll", HB_DYN_CTYPE_VOID, "Llama", cModel, cPrompt, CallBack( { | cStr | AI_FORM.oAnswer.SetFocus(), AI_FORM.oAnswer.Value += cStr } ) )
 

have now change to

Code: Select all  Expand view  RUN

 Llama( "phi-2_Q4_K_M.gguf", RTrim( AI_FORM.cPrompt.Value ),;
   CALLBACK( { | cStr | AI_FORM.cAnswer.SetFocus(), AI_FORM.cAnswer.Value := AI_FORM.cAnswer.Value + cStr } )  )

FUNCTION Llama(cModel,cPrompt, pFunc)
RETURN HMG_CallDLL( "llama64.dll", HB_DYN_CTYPE_VOID, cModel, cPrompt, pFunc)    
 

it does compile and start ... but "no Answer

---

about "llama" ; i have read about CUDA which need external Grafic Card
did somebody get Fivewin Sample running using "internal" Grafic Solution without CUDA

[Jimmy]

command prompt
Run vcvarsall.bat
Change to the directory from step 3 (temp).
Call there cmake .. (parameter is ..) and this will create the sln file for VS.

i did what you told me ... but still have Problem

Microsoft Windows [Version 10.0.19045.3803]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

f:\llama.cpp>call "%ProgramFiles%\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.2.6
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

f:\llama.cpp>
f:\llama.cpp>cmake ..
CMake Error: The source directory "F:/" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

f:\llama.cpp>dir *.txt
Datenträger in Laufwerk F: ist EVO_F
Volumeseriennummer: F4CE-E78C

Verzeichnis von f:\llama.cpp

21.12.2023 04:16 32.658 CMakeLists.txt
21.12.2023 04:16 57 requirements-hf-to-gguf.txt
21.12.2023 04:16 91 requirements.txt
3 Datei(en), 32.806 Bytes
0 Verzeichnis(se), 1.310.322.356.224 Bytes frei

f:\llama.cpp>

have change to

Code: Select all  Expand view  RUN

cMake .

and now got

-- Generating done
-- Build files have been written to: F:/llama.cpp

thx for help

[Jimmy]

hi,

i have change all Property and CODE as advice but can´t create Simple.DLL

Die Neuerstellung wurde um 02:50 gestartet...
1>------ Neues Erstellen gestartet: Projekt: ZERO_CHECK, Konfiguration: Release x64 ------
1>1>Checking Build System
2>------ Neues Erstellen gestartet: Projekt: ggml, Konfiguration: Release x64 ------
3>------ Neues Erstellen gestartet: Projekt: build_info, Konfiguration: Release x64 ------
2>Building Custom Rule F:/llama.cpp/CMakeLists.txt
3>Generating build details from Git
2>ggml.c
3>-- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.43.0.windows.1")
3>Building Custom Rule F:/llama.cpp/common/CMakeLists.txt
3>build-info.cpp
2>C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um\winbase.h(9531,5): warning C5105: Die Makroerweiterung, die "defined" erzeugt, weist ein nicht definiertes Verhalten auf.
2>(Quelldatei „ggml.c“ wird kompiliert)
3>build_info.vcxproj -> F:\llama.cpp\common\build_info.dir\Release\build_info.lib
2>ggml-alloc.c
2>ggml-backend.c
2>F:\llama.cpp\ggml-backend.c(285,14): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "int", Datenverlust möglich
2>ggml-quants.c
2>F:\llama.cpp\ggml-quants.c(627,26): warning C4244: "=": Konvertierung von "float" in "int8_t", möglicher Datenverlust
2>F:\llama.cpp\ggml-quants.c(845,36): warning C4244: "=": Konvertierung von "float" in "int8_t", möglicher Datenverlust
2>F:\llama.cpp\ggml-quants.c(846,36): warning C4244: "=": Konvertierung von "float" in "int8_t", möglicher Datenverlust
2>Code wird generiert...
2>F:\llama.cpp\ggml-backend.c(703): warning C4715: "get_allocr_backend": Nicht alle Codepfade geben einen Wert zurück.
2>ggml.vcxproj -> F:\llama.cpp\ggml.dir\Release\ggml.lib
2>Erstellen des Projekts ggml.vcxproj beendet.
4>------ Neues Erstellen gestartet: Projekt: llama, Konfiguration: Release x64 ------
4>Building Custom Rule F:/llama.cpp/CMakeLists.txt
4>llama.cpp
4>F:\llama.cpp\llama.cpp(1230,31): warning C4305: "Initialisierung": Verkürzung von "double" in "float"
4>F:\llama.cpp\llama.cpp(2778,69): warning C4566: Das durch den universellen Zeichennamen "\u010A" dargestellte Zeichen kann in der aktuellen Codepage (1252) nicht dargestellt werden.
4>F:\llama.cpp\llama.cpp(10420,28): warning C4146: Einem vorzeichenlosen Typ wurde ein unärer Minus-Operator zugewiesen. Das Ergebnis ist weiterhin vorzeichenlos.
4>F:\llama.cpp\llama.cpp(10450,28): warning C4146: Einem vorzeichenlosen Typ wurde ein unärer Minus-Operator zugewiesen. Das Ergebnis ist weiterhin vorzeichenlos.
4>llama.vcxproj -> F:\llama.cpp\Release\llama.lib
4>Erstellen des Projekts llama.vcxproj beendet.
5>------ Neues Erstellen gestartet: Projekt: common, Konfiguration: Release x64 ------
5>Building Custom Rule F:/llama.cpp/common/CMakeLists.txt
5>common.cpp
5>sampling.cpp
5>F:\llama.cpp\common\sampling.cpp(75,45): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "int", Datenverlust möglich
5>F:\llama.cpp\common\sampling.cpp(75,20): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "const int", Datenverlust möglich
5>console.cpp
5>F:\llama.cpp\common\console.cpp(253,30): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "DWORD", Datenverlust möglich
5>F:\llama.cpp\common\console.cpp(407,28): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "int", Datenverlust möglich
5>grammar-parser.cpp
5>train.cpp
5>Code wird generiert...
5>F:\llama.cpp\common\common.cpp(963): warning C4715: "gpt_random_prompt": Nicht alle Codepfade geben einen Wert zurück.
5>common.vcxproj -> F:\llama.cpp\common\Release\common.lib
5>Erstellen des Projekts common.vcxproj beendet.
6>------ Neues Erstellen gestartet: Projekt: simple, Konfiguration: Release x64 ------
6>Building Custom Rule F:/llama.cpp/examples/simple/CMakeLists.txt
6>simple.cpp
6>F:\llama.cpp\examples\simple\simple.cpp(16,31): warning C4305: "=": Verkürzung von "double" in "float"
6>F:\llama.cpp\examples\simple\simple.cpp(61,45): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "int", Datenverlust möglich
6>F:\llama.cpp\examples\simple\simple.cpp(61,24): warning C4267: "Initialisierung": Konvertierung von "size_t" nach "const int", Datenverlust möglich
6>F:\llama.cpp\examples\simple\simple.cpp(64,48): error C2065: "new_token_id": nichtdeklarierter Bezeichner
6>F:\llama.cpp\examples\simple\simple.cpp(90,48): warning C4267: "Argument": Konvertierung von "size_t" nach "llama_pos", Datenverlust möglich
6>Die Erstellung des Projekts "simple.vcxproj" ist abgeschlossen -- FEHLER.
========== Alle neu erstellen: 5 erfolgreich, 1 fehlgeschlagen, 0 übersprungen ==========

need more help please

[Antonio Linares]

Dear Jimmy,

> about "llama" ; i have read about CUDA which need external Grafic Card
> did somebody get Fivewin Sample running using "internal" Grafic Solution without CUDA [/quote]

I have no CUDA (nvidia) on my PC and it works great

[Antonio Linares]

Dear Jimmy,

> 6>F:\llama.cpp\examples\simple\simple.cpp(64,48): error C2065: "new_token_id": nichtdeklarierter Bezeichner

I guess it is this line:
const llama_token new_token_id = llama_sample_token_greedy(ctx, &candidates_p);

I have it here at line 122, how is it possible that you have it at line 64 ?

[Jimmy]

hi Antonio,

I have it here at line 122, how is it possible that you have it at line 64 ?

sorry, it was my Mistake while i have add CODE after wrong Line

now it is running under Fivewin

i still try to run it under HMG ... but it still fail

[Jimmy]

hi,

got Answer from Harbour User Group

Code: Select all  Expand view  RUN

FUNCTION Llama(cModel,cPrompt, pFunc)
LOCAL nErr 
*  nErr := HMG_CallDLL( "llama64.dll", HB_DYN_CTYPE_VOID,  "Llama", cModel, cPrompt, pFunc)    
   nErr := CallDll32 ( "Llama" , "llama64.dll" ,cModel , cPrompt, pFunc )
RETURN .T.

but it does not react same Way like Fivewin Sample
it does not write "Answer" on-fly ... it write it on "End" and all Text in "one Block"

[Antonio Linares]

Dear Jimmy,

glad to know that you got it working using FWH

I think the different behavior with HMG is this:
AI_FORM.oAnswer.Value += cStr

with FWH we use Class TMultiGet METHOD Append( cStr )

Code: Select all  Expand view  RUN

     METHOD Append( cText, nLen ) INLINE ;
                     nLen := ::Len()  ,;
                     SendMessage( ::hWnd, EM_SETSEL, nLen, nLen ),;
                     ::Replace( cText )

[Jimmy]

hi Antonio,

got a Solution

Code: Select all  Expand view  RUN

 Llama( "phi-2_Q4_K_M.gguf", RTrim( AI_FORM.cPrompt.Value ),;
   CALLBACK( { | cStr | AI_FORM.cAnswer.SetFocus ,;
                        IF(cStr=CHR(10),cStr:=CRLF,cStr) ,;
                        AI_FORM.cAnswer.Value := AI_FORM.cAnswer.Value + cStr } )  )
 

i do replace CHR(10) with CRLF and now it look fine

why "Answer" is not shown until "ready" is still a Mistery

[Antonio Linares]

> why "Answer" is not shown until "ready" is still a Mistery

Try with something similar to this:

Code: Select all  Expand view  RUN

{ | cStr | AI_FORM.cAnswer.SetFocus ,;
                        IF(cStr=CHR(10),cStr:=CRLF,cStr) ,;
                        AI_FORM.cAnswer.Value := AI_FORM.cAnswer.Value + cStr, SysRefresh() }