FiveTech Software tech support forums

www.FiveTechSoft.com
https://fivetechsupport.com/forums/

Using Microsoft AI Phi-2 from FWH

https://fivetechsupport.com/forums/viewtopic.php?f=3&t=44028

Page 1 of 4

Using Microsoft AI Phi-2 from FWH

PostPosted: Sun Dec 17, 2023 9:06 am
by Antonio Linares
See the speed, really fine! :-)

This will be included in next FWH build. If you need it before, please send me an email

You don't have to pay OpenAI to have AI on your FWH apps :-)

Image

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Sun Dec 17, 2023 9:11 am
by Antonio Linares
Really easy to be used from our FWH apps:

Code: Select all  Expand view
#include "Fivewin.ch"

function Main()

   local oDlg, cPrompt := PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
   local cAnswer := "", oAnswer, oBtn

   DEFINE DIALOG oDlg SIZE 700, 500 TITLE "FWH AI"

   @ 1, 1 GET cPrompt SIZE 300, 15

   @ 3, 1 GET oAnswer VAR cAnswer MULTILINE SIZE 300, 200

   @ 0.7, 52.5 BUTTON oBtn PROMPT "start" ;
      ACTION ( oBtn:Disable(), Llama( "phi-2_Q4_K_M.gguf", RTrim( cPrompt ),;
               CallBack( { | cStr | oAnswer:SetFocus(), oAnswer:Append( cStr ) } ) ),;
               oBtn:Enable(), oBtn:SetFocus() )

   @ 2.2, 52.5 BUTTON "Clear" ACTION oAnswer:SetText( "" )

   ACTIVATE DIALOG oDlg CENTERED

return nil

DLL FUNCTION Llama( cModel AS LPSTR, cPrompt AS LPSTR, pFunc AS PTR ) AS VOID PASCAL LIB "llama64.dll"

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Sun Dec 17, 2023 11:08 pm
by Jimmy
hi Antomio,

where is ""llama64.dll" come from :?:

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 7:23 am
by Antonio Linares
Dear Jimmy,

For most of GGUF files:
https://github.com/ggerganov/llama.cpp

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 2:56 pm
by Antonio Linares
Dear Jimmy and friends,

Here you have llama64.dll:
https://github.com/FiveTechSoft/FWH_tools/blob/master/llama64.dll

It has been tested using these models. We recommend Phi-2 as it is the smallest and better. Every few days are appearing new and enhanced models :-)

Microsoft Phi-2
https://huggingface.co/kroonen/phi-2-GGUF/resolve/main/phi-2_Q4_K_M.gguf?download=true

Microsoft Orca2
https://huggingface.co/TheBloke/Orca-2-7B-GGUF/resolve/main/orca-2-7b.Q2_K.gguf?download=true

Here you have a working example:
Code: Select all  Expand view
#include "Fivewin.ch"

function Main()

   local oDlg, cPrompt := PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
   local cAnswer := "", oAnswer, oBtn

   DEFINE DIALOG oDlg SIZE 700, 500 TITLE "FWH AI"

   @ 1, 1 GET cPrompt MULTILINE SIZE 300, 15

   @ 3, 1 GET oAnswer VAR cAnswer MULTILINE SIZE 300, 200

   @ 0.7, 52.5 BUTTON oBtn PROMPT "start" ;
      ACTION ( oBtn:Disable(), Llama( "phi-2_Q4_K_M.gguf", RTrim( cPrompt ),;
               CallBack( { | cStr | oAnswer:SetFocus(), oAnswer:Append( cStr ) } ) ),;
               oBtn:Enable(), oBtn:SetFocus() )

   @ 2.2, 52.5 BUTTON "Clear" ACTION oAnswer:SetText( "" )

   ACTIVATE DIALOG oDlg CENTERED

return nil

DLL FUNCTION Llama( cModel AS LPSTR, cPrompt AS LPSTR, pFunc AS PTR ) AS VOID PASCAL LIB "llama64.dll"

#pragma BEGINDUMP

#include <Windows.h>
#include <hbapi.h>
#include <hbapiitm.h>

static PHB_ITEM pBlock;

static void callback( char * szMsg )
{
   PHB_ITEM pStr = hb_itemPutC( NULL, szMsg );
   hb_evalBlock1( pBlock, pStr );
   hb_itemRelease( pStr );
}

HB_FUNC( CALLBACK )
{
   pBlock = hb_gcGripGet( hb_param( 1, HB_IT_BLOCK ) );
   hb_retnll( ( HB_LONGLONG ) callback );
}

#pragma ENDDUMP

Unfortunately the llama32.dll can't handle these AI models cause memory size, so you must build it as a Harbour and FWH 64 bits app and use llama64.dll

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 3:31 pm
by Enrico Maria Giordano
It does not work here. I click on Start and nothing happened. What am I missing? I compiled it with Harbour and FWH64.

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 8:07 pm
by Antonio Linares
Dear Enrico,

Have you downloaded "phi-2_Q4_K_M.gguf" and have you properly provided its path in the PRG code ?

Do you get any message or warning ?

How many ram do you have in your pc ?

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 8:37 pm
by Enrico Maria Giordano
Antonio Linares wrote:Dear Enrico,

Have you downloaded "phi-2_Q4_K_M.gguf" and have you properly provided its path in the PRG code ?


No. :-) Now it works. This is allows it to be compiled with xHarbour too:

Code: Select all  Expand view
hb_retnll( ( LONGLONG ) callback );


A Stop button would be useful. After a certain amount of line of text it seems to freeze. Maybe a scrolling is needed?

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 9:57 pm
by Antonio Linares
Please download llama64.dll again:

https://github.com/FiveTechSoft/FWH_tools/blob/master/llama64.dll

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 10:17 pm
by Antonio Linares
Very interesting:

Fine-tuning Microsoft's Phi-2 on your own data:
https://github.com/brevdev/notebooks/blob/main/phi2-finetune-own-data.ipynb

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Mon Dec 18, 2023 10:42 pm
by Antonio Linares
This version also "speaks" so there is no need to read :-)

Code: Select all  Expand view
#include "Fivewin.ch"

function Main()

   local oDlg, cPrompt := PadR( "List 10 possible uses of AI from my Windows apps.", 200 )
   local cAnswer := "", oAnswer, oBtn
   local oVoice := TOleAuto():New( "Sapi.SPVoice" )

   DEFINE DIALOG oDlg SIZE 700, 500 TITLE "FWH AI"

   @ 1, 1 GET cPrompt MULTILINE SIZE 300, 15

   @ 3, 1 GET oAnswer VAR cAnswer MULTILINE SIZE 300, 200

   @ 0.7, 52.5 BUTTON oBtn PROMPT "start" ;
      ACTION ( oBtn:Disable(), Llama( "phi-2_Q4_K_M.gguf", RTrim( cPrompt ),;
               CallBack( { | cStr | oAnswer:SetFocus(), oAnswer:Append( cStr ) } ) ),;
               oBtn:Enable(), oBtn:SetFocus() )

   @ 2.2, 52.5 BUTTON "Clear" ACTION oAnswer:SetText( "" )

   @ 3.2, 52.5 BUTTON "Speak" ACTION oVoice:Speak( oAnswer:GetText() )

   ACTIVATE DIALOG oDlg CENTERED

return nil

DLL FUNCTION Llama( cModel AS LPSTR, cPrompt AS LPSTR, pFunc AS PTR ) AS VOID PASCAL LIB "llama64.dll"

#pragma BEGINDUMP

#include <Windows.h>
#include <hbapi.h>
#include <hbapiitm.h>

static PHB_ITEM pBlock;

static void callback( char * szMsg )
{
   PHB_ITEM pStr = hb_itemPutC( NULL, szMsg );
   hb_evalBlock1( pBlock, pStr );
   hb_itemRelease( pStr );
}

HB_FUNC( CALLBACK )
{
   pBlock = hb_gcGripGet( hb_param( 1, HB_IT_BLOCK ) );
   hb_retnll( ( HB_LONGLONG ) callback );
}

#pragma ENDDUMP

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Tue Dec 19, 2023 5:01 am
by anserkk
Will this work in FWH 32bit with Harbour and Borland BCC 7 ?

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Tue Dec 19, 2023 7:20 am
by Antonio Linares
Dear Anser,

Unfortunately this only works on 64 bits due to memory requirements as 32 bits is not able to manage memory as efficiently as 64 bits does.

The entire AI model has to be loaded in memory and 32 bits is not able to do it.

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Tue Dec 19, 2023 7:39 am
by Jimmy
hi,

have buildh64 Sample with my *.gguf Path but got Error Message loading llama64.dll :shock:
llama64.dll is in same Folder like EXE

what i´m doing wrong :?:

Re: Using Microsoft AI Phi-2 from FWH

PostPosted: Tue Dec 19, 2023 8:47 am
by Antonio Linares
Dear Jimmy,

Please install this:
https://aka.ms/vs/17/release/vc_redist.x64.exe
All times are UTC
Page 1 of 4
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/